Port of MicroPython to the mbed platform. See micropython-repl for an interactive program.

Dependents:   micropython-repl

This a port of MicroPython to the mbed Classic platform.

This provides an interpreter running on the board's USB serial connection.

Getting Started

Import the micropython-repl program into your IDE workspace on developer.mbed.org. Compile and download to your board. Connect to the USB serial port in your usual manner. You should get a startup message similar to the following:

  MicroPython v1.7-155-gdddcdd8 on 2016-04-23; K64F with ARM
  Type "help()" for more information.
  >>>

Then you can start using micropython. For example:

  >>> from mbed import DigitalOut
  >>> from pins import LED1
  >>> led = DigitalOut(LED1)
  >>> led.write(1)

Requirements

You need approximately 100K of flash memory, so this will be no good for boards with smaller amounts of storage.

Caveats

This can be considered an alpha release of the port; things may not work; APIs may change in later releases. It is NOT an official part part the micropython project, so if anything doesn't work, blame me. If it does work, most of the credit is due to micropython.

  • Only a few of the mbed classes are available in micropython so far, and not all methods of those that are.
  • Only a few boards have their full range of pin names available; for others, only a few standard ones (USBTX, USBRX, LED1) are implemented.
  • The garbage collector is not yet implemented. The interpreter will gradually consume memory and then fail.
  • Exceptions from the mbed classes are not yet handled.
  • Asynchronous processing (e.g. events on inputs) is not supported.

Credits

  • Damien P. George and other contributors who created micropython.
  • Colin Hogben, author of this port.
Committer:
Colin Hogben
Date:
Sat Apr 16 22:43:41 2016 +0100
Revision:
2:c89e95946844
Parent:
0:5868e8752d44
py: update to upstream master

Who changed what in which revision?

UserRevisionLine numberNew contents of line
pythontech 0:5868e8752d44 1 /*
pythontech 0:5868e8752d44 2 * This file is part of the Micro Python project, http://micropython.org/
pythontech 0:5868e8752d44 3 *
pythontech 0:5868e8752d44 4 * The MIT License (MIT)
pythontech 0:5868e8752d44 5 *
pythontech 0:5868e8752d44 6 * Copyright (c) 2013-2015 Damien P. George
pythontech 0:5868e8752d44 7 *
pythontech 0:5868e8752d44 8 * Permission is hereby granted, free of charge, to any person obtaining a copy
pythontech 0:5868e8752d44 9 * of this software and associated documentation files (the "Software"), to deal
pythontech 0:5868e8752d44 10 * in the Software without restriction, including without limitation the rights
pythontech 0:5868e8752d44 11 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
pythontech 0:5868e8752d44 12 * copies of the Software, and to permit persons to whom the Software is
pythontech 0:5868e8752d44 13 * furnished to do so, subject to the following conditions:
pythontech 0:5868e8752d44 14 *
pythontech 0:5868e8752d44 15 * The above copyright notice and this permission notice shall be included in
pythontech 0:5868e8752d44 16 * all copies or substantial portions of the Software.
pythontech 0:5868e8752d44 17 *
pythontech 0:5868e8752d44 18 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
pythontech 0:5868e8752d44 19 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
pythontech 0:5868e8752d44 20 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
pythontech 0:5868e8752d44 21 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
pythontech 0:5868e8752d44 22 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
pythontech 0:5868e8752d44 23 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
pythontech 0:5868e8752d44 24 * THE SOFTWARE.
pythontech 0:5868e8752d44 25 */
pythontech 0:5868e8752d44 26
pythontech 0:5868e8752d44 27 #include <stdbool.h>
pythontech 0:5868e8752d44 28 #include <stdint.h>
pythontech 0:5868e8752d44 29 #include <stdio.h>
pythontech 0:5868e8752d44 30 #include <unistd.h> // for ssize_t
pythontech 0:5868e8752d44 31 #include <assert.h>
pythontech 0:5868e8752d44 32 #include <string.h>
pythontech 0:5868e8752d44 33
pythontech 0:5868e8752d44 34 #include "py/nlr.h"
pythontech 0:5868e8752d44 35 #include "py/lexer.h"
pythontech 0:5868e8752d44 36 #include "py/parse.h"
pythontech 0:5868e8752d44 37 #include "py/parsenum.h"
pythontech 0:5868e8752d44 38 #include "py/runtime0.h"
pythontech 0:5868e8752d44 39 #include "py/runtime.h"
pythontech 0:5868e8752d44 40 #include "py/objint.h"
pythontech 0:5868e8752d44 41 #include "py/builtin.h"
pythontech 0:5868e8752d44 42
pythontech 0:5868e8752d44 43 #if MICROPY_ENABLE_COMPILER
pythontech 0:5868e8752d44 44
pythontech 0:5868e8752d44 45 #define RULE_ACT_ARG_MASK (0x0f)
pythontech 0:5868e8752d44 46 #define RULE_ACT_KIND_MASK (0x30)
pythontech 0:5868e8752d44 47 #define RULE_ACT_ALLOW_IDENT (0x40)
pythontech 0:5868e8752d44 48 #define RULE_ACT_ADD_BLANK (0x80)
pythontech 0:5868e8752d44 49 #define RULE_ACT_OR (0x10)
pythontech 0:5868e8752d44 50 #define RULE_ACT_AND (0x20)
pythontech 0:5868e8752d44 51 #define RULE_ACT_LIST (0x30)
pythontech 0:5868e8752d44 52
pythontech 0:5868e8752d44 53 #define RULE_ARG_KIND_MASK (0xf000)
pythontech 0:5868e8752d44 54 #define RULE_ARG_ARG_MASK (0x0fff)
pythontech 0:5868e8752d44 55 #define RULE_ARG_TOK (0x1000)
pythontech 0:5868e8752d44 56 #define RULE_ARG_RULE (0x2000)
pythontech 0:5868e8752d44 57 #define RULE_ARG_OPT_RULE (0x3000)
pythontech 0:5868e8752d44 58
pythontech 0:5868e8752d44 59 // (un)comment to use rule names; for debugging
pythontech 0:5868e8752d44 60 //#define USE_RULE_NAME (1)
pythontech 0:5868e8752d44 61
pythontech 0:5868e8752d44 62 typedef struct _rule_t {
pythontech 0:5868e8752d44 63 byte rule_id;
pythontech 0:5868e8752d44 64 byte act;
pythontech 0:5868e8752d44 65 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 66 const char *rule_name;
pythontech 0:5868e8752d44 67 #endif
pythontech 0:5868e8752d44 68 uint16_t arg[];
pythontech 0:5868e8752d44 69 } rule_t;
pythontech 0:5868e8752d44 70
pythontech 0:5868e8752d44 71 enum {
pythontech 0:5868e8752d44 72 #define DEF_RULE(rule, comp, kind, ...) RULE_##rule,
pythontech 0:5868e8752d44 73 #include "py/grammar.h"
pythontech 0:5868e8752d44 74 #undef DEF_RULE
pythontech 0:5868e8752d44 75 RULE_maximum_number_of,
pythontech 0:5868e8752d44 76 RULE_string, // special node for non-interned string
pythontech 0:5868e8752d44 77 RULE_bytes, // special node for non-interned bytes
pythontech 0:5868e8752d44 78 RULE_const_object, // special node for a constant, generic Python object
pythontech 0:5868e8752d44 79 };
pythontech 0:5868e8752d44 80
pythontech 0:5868e8752d44 81 #define or(n) (RULE_ACT_OR | n)
pythontech 0:5868e8752d44 82 #define and(n) (RULE_ACT_AND | n)
Colin Hogben 2:c89e95946844 83 #define and_ident(n) (RULE_ACT_AND | n | RULE_ACT_ALLOW_IDENT)
Colin Hogben 2:c89e95946844 84 #define and_blank(n) (RULE_ACT_AND | n | RULE_ACT_ADD_BLANK)
pythontech 0:5868e8752d44 85 #define one_or_more (RULE_ACT_LIST | 2)
pythontech 0:5868e8752d44 86 #define list (RULE_ACT_LIST | 1)
pythontech 0:5868e8752d44 87 #define list_with_end (RULE_ACT_LIST | 3)
pythontech 0:5868e8752d44 88 #define tok(t) (RULE_ARG_TOK | MP_TOKEN_##t)
pythontech 0:5868e8752d44 89 #define rule(r) (RULE_ARG_RULE | RULE_##r)
pythontech 0:5868e8752d44 90 #define opt_rule(r) (RULE_ARG_OPT_RULE | RULE_##r)
pythontech 0:5868e8752d44 91 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 92 #define DEF_RULE(rule, comp, kind, ...) static const rule_t rule_##rule = { RULE_##rule, kind, #rule, { __VA_ARGS__ } };
pythontech 0:5868e8752d44 93 #else
pythontech 0:5868e8752d44 94 #define DEF_RULE(rule, comp, kind, ...) static const rule_t rule_##rule = { RULE_##rule, kind, { __VA_ARGS__ } };
pythontech 0:5868e8752d44 95 #endif
pythontech 0:5868e8752d44 96 #include "py/grammar.h"
pythontech 0:5868e8752d44 97 #undef or
pythontech 0:5868e8752d44 98 #undef and
pythontech 0:5868e8752d44 99 #undef list
pythontech 0:5868e8752d44 100 #undef list_with_end
pythontech 0:5868e8752d44 101 #undef tok
pythontech 0:5868e8752d44 102 #undef rule
pythontech 0:5868e8752d44 103 #undef opt_rule
pythontech 0:5868e8752d44 104 #undef one_or_more
pythontech 0:5868e8752d44 105 #undef DEF_RULE
pythontech 0:5868e8752d44 106
pythontech 0:5868e8752d44 107 STATIC const rule_t *rules[] = {
pythontech 0:5868e8752d44 108 #define DEF_RULE(rule, comp, kind, ...) &rule_##rule,
pythontech 0:5868e8752d44 109 #include "py/grammar.h"
pythontech 0:5868e8752d44 110 #undef DEF_RULE
pythontech 0:5868e8752d44 111 };
pythontech 0:5868e8752d44 112
pythontech 0:5868e8752d44 113 typedef struct _rule_stack_t {
pythontech 0:5868e8752d44 114 size_t src_line : 8 * sizeof(size_t) - 8; // maximum bits storing source line number
pythontech 0:5868e8752d44 115 size_t rule_id : 8; // this must be large enough to fit largest rule number
pythontech 0:5868e8752d44 116 size_t arg_i; // this dictates the maximum nodes in a "list" of things
pythontech 0:5868e8752d44 117 } rule_stack_t;
pythontech 0:5868e8752d44 118
pythontech 0:5868e8752d44 119 typedef struct _mp_parse_chunk_t {
pythontech 0:5868e8752d44 120 size_t alloc;
pythontech 0:5868e8752d44 121 union {
pythontech 0:5868e8752d44 122 size_t used;
pythontech 0:5868e8752d44 123 struct _mp_parse_chunk_t *next;
pythontech 0:5868e8752d44 124 } union_;
pythontech 0:5868e8752d44 125 byte data[];
pythontech 0:5868e8752d44 126 } mp_parse_chunk_t;
pythontech 0:5868e8752d44 127
pythontech 0:5868e8752d44 128 typedef enum {
pythontech 0:5868e8752d44 129 PARSE_ERROR_NONE = 0,
pythontech 0:5868e8752d44 130 PARSE_ERROR_MEMORY,
pythontech 0:5868e8752d44 131 PARSE_ERROR_CONST,
pythontech 0:5868e8752d44 132 } parse_error_t;
pythontech 0:5868e8752d44 133
pythontech 0:5868e8752d44 134 typedef struct _parser_t {
pythontech 0:5868e8752d44 135 parse_error_t parse_error;
pythontech 0:5868e8752d44 136
pythontech 0:5868e8752d44 137 size_t rule_stack_alloc;
pythontech 0:5868e8752d44 138 size_t rule_stack_top;
pythontech 0:5868e8752d44 139 rule_stack_t *rule_stack;
pythontech 0:5868e8752d44 140
pythontech 0:5868e8752d44 141 size_t result_stack_alloc;
pythontech 0:5868e8752d44 142 size_t result_stack_top;
pythontech 0:5868e8752d44 143 mp_parse_node_t *result_stack;
pythontech 0:5868e8752d44 144
pythontech 0:5868e8752d44 145 mp_lexer_t *lexer;
pythontech 0:5868e8752d44 146
pythontech 0:5868e8752d44 147 mp_parse_tree_t tree;
pythontech 0:5868e8752d44 148 mp_parse_chunk_t *cur_chunk;
pythontech 0:5868e8752d44 149
pythontech 0:5868e8752d44 150 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 151 mp_map_t consts;
pythontech 0:5868e8752d44 152 #endif
pythontech 0:5868e8752d44 153 } parser_t;
pythontech 0:5868e8752d44 154
pythontech 0:5868e8752d44 155 STATIC void *parser_alloc(parser_t *parser, size_t num_bytes) {
pythontech 0:5868e8752d44 156 // use a custom memory allocator to store parse nodes sequentially in large chunks
pythontech 0:5868e8752d44 157
pythontech 0:5868e8752d44 158 mp_parse_chunk_t *chunk = parser->cur_chunk;
pythontech 0:5868e8752d44 159
pythontech 0:5868e8752d44 160 if (chunk != NULL && chunk->union_.used + num_bytes > chunk->alloc) {
pythontech 0:5868e8752d44 161 // not enough room at end of previously allocated chunk so try to grow
pythontech 0:5868e8752d44 162 mp_parse_chunk_t *new_data = (mp_parse_chunk_t*)m_renew_maybe(byte, chunk,
pythontech 0:5868e8752d44 163 sizeof(mp_parse_chunk_t) + chunk->alloc,
pythontech 0:5868e8752d44 164 sizeof(mp_parse_chunk_t) + chunk->alloc + num_bytes, false);
pythontech 0:5868e8752d44 165 if (new_data == NULL) {
pythontech 0:5868e8752d44 166 // could not grow existing memory; shrink it to fit previous
pythontech 0:5868e8752d44 167 (void)m_renew_maybe(byte, chunk, sizeof(mp_parse_chunk_t) + chunk->alloc,
pythontech 0:5868e8752d44 168 sizeof(mp_parse_chunk_t) + chunk->union_.used, false);
pythontech 0:5868e8752d44 169 chunk->alloc = chunk->union_.used;
pythontech 0:5868e8752d44 170 chunk->union_.next = parser->tree.chunk;
pythontech 0:5868e8752d44 171 parser->tree.chunk = chunk;
pythontech 0:5868e8752d44 172 chunk = NULL;
pythontech 0:5868e8752d44 173 } else {
pythontech 0:5868e8752d44 174 // could grow existing memory
pythontech 0:5868e8752d44 175 chunk->alloc += num_bytes;
pythontech 0:5868e8752d44 176 }
pythontech 0:5868e8752d44 177 }
pythontech 0:5868e8752d44 178
pythontech 0:5868e8752d44 179 if (chunk == NULL) {
pythontech 0:5868e8752d44 180 // no previous chunk, allocate a new chunk
pythontech 0:5868e8752d44 181 size_t alloc = MICROPY_ALLOC_PARSE_CHUNK_INIT;
pythontech 0:5868e8752d44 182 if (alloc < num_bytes) {
pythontech 0:5868e8752d44 183 alloc = num_bytes;
pythontech 0:5868e8752d44 184 }
pythontech 0:5868e8752d44 185 chunk = (mp_parse_chunk_t*)m_new(byte, sizeof(mp_parse_chunk_t) + alloc);
pythontech 0:5868e8752d44 186 chunk->alloc = alloc;
pythontech 0:5868e8752d44 187 chunk->union_.used = 0;
pythontech 0:5868e8752d44 188 parser->cur_chunk = chunk;
pythontech 0:5868e8752d44 189 }
pythontech 0:5868e8752d44 190
pythontech 0:5868e8752d44 191 byte *ret = chunk->data + chunk->union_.used;
pythontech 0:5868e8752d44 192 chunk->union_.used += num_bytes;
pythontech 0:5868e8752d44 193 return ret;
pythontech 0:5868e8752d44 194 }
pythontech 0:5868e8752d44 195
pythontech 0:5868e8752d44 196 STATIC void push_rule(parser_t *parser, size_t src_line, const rule_t *rule, size_t arg_i) {
pythontech 0:5868e8752d44 197 if (parser->parse_error) {
pythontech 0:5868e8752d44 198 return;
pythontech 0:5868e8752d44 199 }
pythontech 0:5868e8752d44 200 if (parser->rule_stack_top >= parser->rule_stack_alloc) {
pythontech 0:5868e8752d44 201 rule_stack_t *rs = m_renew_maybe(rule_stack_t, parser->rule_stack, parser->rule_stack_alloc, parser->rule_stack_alloc + MICROPY_ALLOC_PARSE_RULE_INC, true);
pythontech 0:5868e8752d44 202 if (rs == NULL) {
pythontech 0:5868e8752d44 203 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 204 return;
pythontech 0:5868e8752d44 205 }
pythontech 0:5868e8752d44 206 parser->rule_stack = rs;
pythontech 0:5868e8752d44 207 parser->rule_stack_alloc += MICROPY_ALLOC_PARSE_RULE_INC;
pythontech 0:5868e8752d44 208 }
pythontech 0:5868e8752d44 209 rule_stack_t *rs = &parser->rule_stack[parser->rule_stack_top++];
pythontech 0:5868e8752d44 210 rs->src_line = src_line;
pythontech 0:5868e8752d44 211 rs->rule_id = rule->rule_id;
pythontech 0:5868e8752d44 212 rs->arg_i = arg_i;
pythontech 0:5868e8752d44 213 }
pythontech 0:5868e8752d44 214
pythontech 0:5868e8752d44 215 STATIC void push_rule_from_arg(parser_t *parser, size_t arg) {
pythontech 0:5868e8752d44 216 assert((arg & RULE_ARG_KIND_MASK) == RULE_ARG_RULE || (arg & RULE_ARG_KIND_MASK) == RULE_ARG_OPT_RULE);
pythontech 0:5868e8752d44 217 size_t rule_id = arg & RULE_ARG_ARG_MASK;
pythontech 0:5868e8752d44 218 assert(rule_id < RULE_maximum_number_of);
pythontech 0:5868e8752d44 219 push_rule(parser, parser->lexer->tok_line, rules[rule_id], 0);
pythontech 0:5868e8752d44 220 }
pythontech 0:5868e8752d44 221
pythontech 0:5868e8752d44 222 STATIC void pop_rule(parser_t *parser, const rule_t **rule, size_t *arg_i, size_t *src_line) {
pythontech 0:5868e8752d44 223 assert(!parser->parse_error);
pythontech 0:5868e8752d44 224 parser->rule_stack_top -= 1;
pythontech 0:5868e8752d44 225 *rule = rules[parser->rule_stack[parser->rule_stack_top].rule_id];
pythontech 0:5868e8752d44 226 *arg_i = parser->rule_stack[parser->rule_stack_top].arg_i;
pythontech 0:5868e8752d44 227 *src_line = parser->rule_stack[parser->rule_stack_top].src_line;
pythontech 0:5868e8752d44 228 }
pythontech 0:5868e8752d44 229
pythontech 0:5868e8752d44 230 mp_parse_node_t mp_parse_node_new_leaf(size_t kind, mp_int_t arg) {
pythontech 0:5868e8752d44 231 if (kind == MP_PARSE_NODE_SMALL_INT) {
pythontech 0:5868e8752d44 232 return (mp_parse_node_t)(kind | (arg << 1));
pythontech 0:5868e8752d44 233 }
pythontech 0:5868e8752d44 234 return (mp_parse_node_t)(kind | (arg << 4));
pythontech 0:5868e8752d44 235 }
pythontech 0:5868e8752d44 236
pythontech 0:5868e8752d44 237 bool mp_parse_node_get_int_maybe(mp_parse_node_t pn, mp_obj_t *o) {
pythontech 0:5868e8752d44 238 if (MP_PARSE_NODE_IS_SMALL_INT(pn)) {
pythontech 0:5868e8752d44 239 *o = MP_OBJ_NEW_SMALL_INT(MP_PARSE_NODE_LEAF_SMALL_INT(pn));
pythontech 0:5868e8752d44 240 return true;
pythontech 0:5868e8752d44 241 } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, RULE_const_object)) {
pythontech 0:5868e8752d44 242 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn;
pythontech 0:5868e8752d44 243 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 244 // nodes are 32-bit pointers, but need to extract 64-bit object
pythontech 0:5868e8752d44 245 *o = (uint64_t)pns->nodes[0] | ((uint64_t)pns->nodes[1] << 32);
pythontech 0:5868e8752d44 246 #else
pythontech 0:5868e8752d44 247 *o = (mp_obj_t)pns->nodes[0];
pythontech 0:5868e8752d44 248 #endif
pythontech 0:5868e8752d44 249 return MP_OBJ_IS_INT(*o);
pythontech 0:5868e8752d44 250 } else {
pythontech 0:5868e8752d44 251 return false;
pythontech 0:5868e8752d44 252 }
pythontech 0:5868e8752d44 253 }
pythontech 0:5868e8752d44 254
pythontech 0:5868e8752d44 255 int mp_parse_node_extract_list(mp_parse_node_t *pn, size_t pn_kind, mp_parse_node_t **nodes) {
pythontech 0:5868e8752d44 256 if (MP_PARSE_NODE_IS_NULL(*pn)) {
pythontech 0:5868e8752d44 257 *nodes = NULL;
pythontech 0:5868e8752d44 258 return 0;
pythontech 0:5868e8752d44 259 } else if (MP_PARSE_NODE_IS_LEAF(*pn)) {
pythontech 0:5868e8752d44 260 *nodes = pn;
pythontech 0:5868e8752d44 261 return 1;
pythontech 0:5868e8752d44 262 } else {
pythontech 0:5868e8752d44 263 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)(*pn);
pythontech 0:5868e8752d44 264 if (MP_PARSE_NODE_STRUCT_KIND(pns) != pn_kind) {
pythontech 0:5868e8752d44 265 *nodes = pn;
pythontech 0:5868e8752d44 266 return 1;
pythontech 0:5868e8752d44 267 } else {
pythontech 0:5868e8752d44 268 *nodes = pns->nodes;
pythontech 0:5868e8752d44 269 return MP_PARSE_NODE_STRUCT_NUM_NODES(pns);
pythontech 0:5868e8752d44 270 }
pythontech 0:5868e8752d44 271 }
pythontech 0:5868e8752d44 272 }
pythontech 0:5868e8752d44 273
pythontech 0:5868e8752d44 274 #if MICROPY_DEBUG_PRINTERS
pythontech 0:5868e8752d44 275 void mp_parse_node_print(mp_parse_node_t pn, size_t indent) {
pythontech 0:5868e8752d44 276 if (MP_PARSE_NODE_IS_STRUCT(pn)) {
pythontech 0:5868e8752d44 277 printf("[% 4d] ", (int)((mp_parse_node_struct_t*)pn)->source_line);
pythontech 0:5868e8752d44 278 } else {
pythontech 0:5868e8752d44 279 printf(" ");
pythontech 0:5868e8752d44 280 }
pythontech 0:5868e8752d44 281 for (size_t i = 0; i < indent; i++) {
pythontech 0:5868e8752d44 282 printf(" ");
pythontech 0:5868e8752d44 283 }
pythontech 0:5868e8752d44 284 if (MP_PARSE_NODE_IS_NULL(pn)) {
pythontech 0:5868e8752d44 285 printf("NULL\n");
pythontech 0:5868e8752d44 286 } else if (MP_PARSE_NODE_IS_SMALL_INT(pn)) {
pythontech 0:5868e8752d44 287 mp_int_t arg = MP_PARSE_NODE_LEAF_SMALL_INT(pn);
pythontech 0:5868e8752d44 288 printf("int(" INT_FMT ")\n", arg);
pythontech 0:5868e8752d44 289 } else if (MP_PARSE_NODE_IS_LEAF(pn)) {
pythontech 0:5868e8752d44 290 uintptr_t arg = MP_PARSE_NODE_LEAF_ARG(pn);
pythontech 0:5868e8752d44 291 switch (MP_PARSE_NODE_LEAF_KIND(pn)) {
pythontech 0:5868e8752d44 292 case MP_PARSE_NODE_ID: printf("id(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 293 case MP_PARSE_NODE_STRING: printf("str(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 294 case MP_PARSE_NODE_BYTES: printf("bytes(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 295 case MP_PARSE_NODE_TOKEN: printf("tok(%u)\n", (uint)arg); break;
pythontech 0:5868e8752d44 296 default: assert(0);
pythontech 0:5868e8752d44 297 }
pythontech 0:5868e8752d44 298 } else {
pythontech 0:5868e8752d44 299 // node must be a mp_parse_node_struct_t
pythontech 0:5868e8752d44 300 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn;
pythontech 0:5868e8752d44 301 if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_string) {
pythontech 0:5868e8752d44 302 printf("literal str(%.*s)\n", (int)pns->nodes[1], (char*)pns->nodes[0]);
pythontech 0:5868e8752d44 303 } else if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_bytes) {
pythontech 0:5868e8752d44 304 printf("literal bytes(%.*s)\n", (int)pns->nodes[1], (char*)pns->nodes[0]);
pythontech 0:5868e8752d44 305 } else if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_const_object) {
pythontech 0:5868e8752d44 306 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 307 printf("literal const(%016llx)\n", (uint64_t)pns->nodes[0] | ((uint64_t)pns->nodes[1] << 32));
pythontech 0:5868e8752d44 308 #else
pythontech 0:5868e8752d44 309 printf("literal const(%p)\n", (mp_obj_t)pns->nodes[0]);
pythontech 0:5868e8752d44 310 #endif
pythontech 0:5868e8752d44 311 } else {
pythontech 0:5868e8752d44 312 size_t n = MP_PARSE_NODE_STRUCT_NUM_NODES(pns);
pythontech 0:5868e8752d44 313 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 314 printf("%s(%u) (n=%u)\n", rules[MP_PARSE_NODE_STRUCT_KIND(pns)]->rule_name, (uint)MP_PARSE_NODE_STRUCT_KIND(pns), (uint)n);
pythontech 0:5868e8752d44 315 #else
pythontech 0:5868e8752d44 316 printf("rule(%u) (n=%u)\n", (uint)MP_PARSE_NODE_STRUCT_KIND(pns), (uint)n);
pythontech 0:5868e8752d44 317 #endif
pythontech 0:5868e8752d44 318 for (size_t i = 0; i < n; i++) {
pythontech 0:5868e8752d44 319 mp_parse_node_print(pns->nodes[i], indent + 2);
pythontech 0:5868e8752d44 320 }
pythontech 0:5868e8752d44 321 }
pythontech 0:5868e8752d44 322 }
pythontech 0:5868e8752d44 323 }
pythontech 0:5868e8752d44 324 #endif // MICROPY_DEBUG_PRINTERS
pythontech 0:5868e8752d44 325
pythontech 0:5868e8752d44 326 /*
pythontech 0:5868e8752d44 327 STATIC void result_stack_show(parser_t *parser) {
pythontech 0:5868e8752d44 328 printf("result stack, most recent first\n");
pythontech 0:5868e8752d44 329 for (ssize_t i = parser->result_stack_top - 1; i >= 0; i--) {
pythontech 0:5868e8752d44 330 mp_parse_node_print(parser->result_stack[i], 0);
pythontech 0:5868e8752d44 331 }
pythontech 0:5868e8752d44 332 }
pythontech 0:5868e8752d44 333 */
pythontech 0:5868e8752d44 334
pythontech 0:5868e8752d44 335 STATIC mp_parse_node_t pop_result(parser_t *parser) {
pythontech 0:5868e8752d44 336 if (parser->parse_error) {
pythontech 0:5868e8752d44 337 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 338 }
pythontech 0:5868e8752d44 339 assert(parser->result_stack_top > 0);
pythontech 0:5868e8752d44 340 return parser->result_stack[--parser->result_stack_top];
pythontech 0:5868e8752d44 341 }
pythontech 0:5868e8752d44 342
pythontech 0:5868e8752d44 343 STATIC mp_parse_node_t peek_result(parser_t *parser, size_t pos) {
pythontech 0:5868e8752d44 344 if (parser->parse_error) {
pythontech 0:5868e8752d44 345 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 346 }
pythontech 0:5868e8752d44 347 assert(parser->result_stack_top > pos);
pythontech 0:5868e8752d44 348 return parser->result_stack[parser->result_stack_top - 1 - pos];
pythontech 0:5868e8752d44 349 }
pythontech 0:5868e8752d44 350
pythontech 0:5868e8752d44 351 STATIC void push_result_node(parser_t *parser, mp_parse_node_t pn) {
pythontech 0:5868e8752d44 352 if (parser->parse_error) {
pythontech 0:5868e8752d44 353 return;
pythontech 0:5868e8752d44 354 }
pythontech 0:5868e8752d44 355 if (parser->result_stack_top >= parser->result_stack_alloc) {
pythontech 0:5868e8752d44 356 mp_parse_node_t *stack = m_renew_maybe(mp_parse_node_t, parser->result_stack, parser->result_stack_alloc, parser->result_stack_alloc + MICROPY_ALLOC_PARSE_RESULT_INC, true);
pythontech 0:5868e8752d44 357 if (stack == NULL) {
pythontech 0:5868e8752d44 358 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 359 return;
pythontech 0:5868e8752d44 360 }
pythontech 0:5868e8752d44 361 parser->result_stack = stack;
pythontech 0:5868e8752d44 362 parser->result_stack_alloc += MICROPY_ALLOC_PARSE_RESULT_INC;
pythontech 0:5868e8752d44 363 }
pythontech 0:5868e8752d44 364 parser->result_stack[parser->result_stack_top++] = pn;
pythontech 0:5868e8752d44 365 }
pythontech 0:5868e8752d44 366
pythontech 0:5868e8752d44 367 STATIC mp_parse_node_t make_node_string_bytes(parser_t *parser, size_t src_line, size_t rule_kind, const char *str, size_t len) {
pythontech 0:5868e8752d44 368 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_parse_node_t) * 2);
pythontech 0:5868e8752d44 369 if (pn == NULL) {
pythontech 0:5868e8752d44 370 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 371 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 372 }
pythontech 0:5868e8752d44 373 pn->source_line = src_line;
pythontech 0:5868e8752d44 374 pn->kind_num_nodes = rule_kind | (2 << 8);
pythontech 0:5868e8752d44 375 char *p = m_new(char, len);
pythontech 0:5868e8752d44 376 memcpy(p, str, len);
pythontech 0:5868e8752d44 377 pn->nodes[0] = (uintptr_t)p;
pythontech 0:5868e8752d44 378 pn->nodes[1] = len;
pythontech 0:5868e8752d44 379 return (mp_parse_node_t)pn;
pythontech 0:5868e8752d44 380 }
pythontech 0:5868e8752d44 381
pythontech 0:5868e8752d44 382 STATIC mp_parse_node_t make_node_const_object(parser_t *parser, size_t src_line, mp_obj_t obj) {
pythontech 0:5868e8752d44 383 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_obj_t));
pythontech 0:5868e8752d44 384 if (pn == NULL) {
pythontech 0:5868e8752d44 385 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 386 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 387 }
pythontech 0:5868e8752d44 388 pn->source_line = src_line;
pythontech 0:5868e8752d44 389 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 390 // nodes are 32-bit pointers, but need to store 64-bit object
pythontech 0:5868e8752d44 391 pn->kind_num_nodes = RULE_const_object | (2 << 8);
pythontech 0:5868e8752d44 392 pn->nodes[0] = (uint64_t)obj;
pythontech 0:5868e8752d44 393 pn->nodes[1] = (uint64_t)obj >> 32;
pythontech 0:5868e8752d44 394 #else
pythontech 0:5868e8752d44 395 pn->kind_num_nodes = RULE_const_object | (1 << 8);
pythontech 0:5868e8752d44 396 pn->nodes[0] = (uintptr_t)obj;
pythontech 0:5868e8752d44 397 #endif
pythontech 0:5868e8752d44 398 return (mp_parse_node_t)pn;
pythontech 0:5868e8752d44 399 }
pythontech 0:5868e8752d44 400
pythontech 0:5868e8752d44 401 STATIC void push_result_token(parser_t *parser) {
pythontech 0:5868e8752d44 402 mp_parse_node_t pn;
pythontech 0:5868e8752d44 403 mp_lexer_t *lex = parser->lexer;
pythontech 0:5868e8752d44 404 if (lex->tok_kind == MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 405 qstr id = qstr_from_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 406 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 407 // lookup identifier in table of dynamic constants
pythontech 0:5868e8752d44 408 mp_map_elem_t *elem = mp_map_lookup(&parser->consts, MP_OBJ_NEW_QSTR(id), MP_MAP_LOOKUP);
pythontech 0:5868e8752d44 409 if (elem != NULL) {
pythontech 0:5868e8752d44 410 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(elem->value));
pythontech 0:5868e8752d44 411 } else
pythontech 0:5868e8752d44 412 #endif
pythontech 0:5868e8752d44 413 {
pythontech 0:5868e8752d44 414 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_ID, id);
pythontech 0:5868e8752d44 415 }
pythontech 0:5868e8752d44 416 } else if (lex->tok_kind == MP_TOKEN_INTEGER) {
pythontech 0:5868e8752d44 417 mp_obj_t o = mp_parse_num_integer(lex->vstr.buf, lex->vstr.len, 0, lex);
pythontech 0:5868e8752d44 418 if (MP_OBJ_IS_SMALL_INT(o)) {
pythontech 0:5868e8752d44 419 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(o));
pythontech 0:5868e8752d44 420 } else {
pythontech 0:5868e8752d44 421 pn = make_node_const_object(parser, lex->tok_line, o);
pythontech 0:5868e8752d44 422 }
pythontech 0:5868e8752d44 423 } else if (lex->tok_kind == MP_TOKEN_FLOAT_OR_IMAG) {
pythontech 0:5868e8752d44 424 mp_obj_t o = mp_parse_num_decimal(lex->vstr.buf, lex->vstr.len, true, false, lex);
pythontech 0:5868e8752d44 425 pn = make_node_const_object(parser, lex->tok_line, o);
pythontech 0:5868e8752d44 426 } else if (lex->tok_kind == MP_TOKEN_STRING || lex->tok_kind == MP_TOKEN_BYTES) {
pythontech 0:5868e8752d44 427 // Don't automatically intern all strings/bytes. doc strings (which are usually large)
pythontech 0:5868e8752d44 428 // will be discarded by the compiler, and so we shouldn't intern them.
pythontech 0:5868e8752d44 429 qstr qst = MP_QSTR_NULL;
pythontech 0:5868e8752d44 430 if (lex->vstr.len <= MICROPY_ALLOC_PARSE_INTERN_STRING_LEN) {
pythontech 0:5868e8752d44 431 // intern short strings
pythontech 0:5868e8752d44 432 qst = qstr_from_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 433 } else {
pythontech 0:5868e8752d44 434 // check if this string is already interned
pythontech 0:5868e8752d44 435 qst = qstr_find_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 436 }
pythontech 0:5868e8752d44 437 if (qst != MP_QSTR_NULL) {
pythontech 0:5868e8752d44 438 // qstr exists, make a leaf node
pythontech 0:5868e8752d44 439 pn = mp_parse_node_new_leaf(lex->tok_kind == MP_TOKEN_STRING ? MP_PARSE_NODE_STRING : MP_PARSE_NODE_BYTES, qst);
pythontech 0:5868e8752d44 440 } else {
pythontech 0:5868e8752d44 441 // not interned, make a node holding a pointer to the string/bytes data
pythontech 0:5868e8752d44 442 pn = make_node_string_bytes(parser, lex->tok_line, lex->tok_kind == MP_TOKEN_STRING ? RULE_string : RULE_bytes, lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 443 }
pythontech 0:5868e8752d44 444 } else {
pythontech 0:5868e8752d44 445 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_TOKEN, lex->tok_kind);
pythontech 0:5868e8752d44 446 }
pythontech 0:5868e8752d44 447 push_result_node(parser, pn);
pythontech 0:5868e8752d44 448 }
pythontech 0:5868e8752d44 449
pythontech 0:5868e8752d44 450 #if MICROPY_COMP_MODULE_CONST
pythontech 0:5868e8752d44 451 STATIC const mp_rom_map_elem_t mp_constants_table[] = {
pythontech 0:5868e8752d44 452 #if MICROPY_PY_UCTYPES
pythontech 0:5868e8752d44 453 { MP_ROM_QSTR(MP_QSTR_uctypes), MP_ROM_PTR(&mp_module_uctypes) },
pythontech 0:5868e8752d44 454 #endif
pythontech 0:5868e8752d44 455 // Extra constants as defined by a port
pythontech 0:5868e8752d44 456 MICROPY_PORT_CONSTANTS
pythontech 0:5868e8752d44 457 };
pythontech 0:5868e8752d44 458 STATIC MP_DEFINE_CONST_MAP(mp_constants_map, mp_constants_table);
pythontech 0:5868e8752d44 459 #endif
pythontech 0:5868e8752d44 460
pythontech 0:5868e8752d44 461 #if MICROPY_COMP_CONST_FOLDING
pythontech 0:5868e8752d44 462 STATIC bool fold_constants(parser_t *parser, const rule_t *rule, size_t num_args) {
pythontech 0:5868e8752d44 463 // this code does folding of arbitrary integer expressions, eg 1 + 2 * 3 + 4
pythontech 0:5868e8752d44 464 // it does not do partial folding, eg 1 + 2 + x -> 3 + x
pythontech 0:5868e8752d44 465
pythontech 0:5868e8752d44 466 mp_obj_t arg0;
pythontech 0:5868e8752d44 467 if (rule->rule_id == RULE_expr
pythontech 0:5868e8752d44 468 || rule->rule_id == RULE_xor_expr
pythontech 0:5868e8752d44 469 || rule->rule_id == RULE_and_expr) {
pythontech 0:5868e8752d44 470 // folding for binary ops: | ^ &
pythontech 0:5868e8752d44 471 mp_parse_node_t pn = peek_result(parser, num_args - 1);
pythontech 0:5868e8752d44 472 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 473 return false;
pythontech 0:5868e8752d44 474 }
pythontech 0:5868e8752d44 475 mp_binary_op_t op;
pythontech 0:5868e8752d44 476 if (rule->rule_id == RULE_expr) {
pythontech 0:5868e8752d44 477 op = MP_BINARY_OP_OR;
pythontech 0:5868e8752d44 478 } else if (rule->rule_id == RULE_xor_expr) {
pythontech 0:5868e8752d44 479 op = MP_BINARY_OP_XOR;
pythontech 0:5868e8752d44 480 } else {
pythontech 0:5868e8752d44 481 op = MP_BINARY_OP_AND;
pythontech 0:5868e8752d44 482 }
pythontech 0:5868e8752d44 483 for (ssize_t i = num_args - 2; i >= 0; --i) {
pythontech 0:5868e8752d44 484 pn = peek_result(parser, i);
pythontech 0:5868e8752d44 485 mp_obj_t arg1;
pythontech 0:5868e8752d44 486 if (!mp_parse_node_get_int_maybe(pn, &arg1)) {
pythontech 0:5868e8752d44 487 return false;
pythontech 0:5868e8752d44 488 }
pythontech 0:5868e8752d44 489 arg0 = mp_binary_op(op, arg0, arg1);
pythontech 0:5868e8752d44 490 }
pythontech 0:5868e8752d44 491 } else if (rule->rule_id == RULE_shift_expr
pythontech 0:5868e8752d44 492 || rule->rule_id == RULE_arith_expr
pythontech 0:5868e8752d44 493 || rule->rule_id == RULE_term) {
pythontech 0:5868e8752d44 494 // folding for binary ops: << >> + - * / % //
pythontech 0:5868e8752d44 495 mp_parse_node_t pn = peek_result(parser, num_args - 1);
pythontech 0:5868e8752d44 496 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 497 return false;
pythontech 0:5868e8752d44 498 }
pythontech 0:5868e8752d44 499 for (ssize_t i = num_args - 2; i >= 1; i -= 2) {
pythontech 0:5868e8752d44 500 pn = peek_result(parser, i - 1);
pythontech 0:5868e8752d44 501 mp_obj_t arg1;
pythontech 0:5868e8752d44 502 if (!mp_parse_node_get_int_maybe(pn, &arg1)) {
pythontech 0:5868e8752d44 503 return false;
pythontech 0:5868e8752d44 504 }
pythontech 0:5868e8752d44 505 mp_token_kind_t tok = MP_PARSE_NODE_LEAF_ARG(peek_result(parser, i));
pythontech 0:5868e8752d44 506 static const uint8_t token_to_op[] = {
pythontech 0:5868e8752d44 507 MP_BINARY_OP_ADD,
pythontech 0:5868e8752d44 508 MP_BINARY_OP_SUBTRACT,
pythontech 0:5868e8752d44 509 MP_BINARY_OP_MULTIPLY,
pythontech 0:5868e8752d44 510 255,//MP_BINARY_OP_POWER,
pythontech 0:5868e8752d44 511 255,//MP_BINARY_OP_TRUE_DIVIDE,
pythontech 0:5868e8752d44 512 MP_BINARY_OP_FLOOR_DIVIDE,
pythontech 0:5868e8752d44 513 MP_BINARY_OP_MODULO,
pythontech 0:5868e8752d44 514 255,//MP_BINARY_OP_LESS
pythontech 0:5868e8752d44 515 MP_BINARY_OP_LSHIFT,
pythontech 0:5868e8752d44 516 255,//MP_BINARY_OP_MORE
pythontech 0:5868e8752d44 517 MP_BINARY_OP_RSHIFT,
pythontech 0:5868e8752d44 518 };
pythontech 0:5868e8752d44 519 mp_binary_op_t op = token_to_op[tok - MP_TOKEN_OP_PLUS];
pythontech 0:5868e8752d44 520 if (op == (mp_binary_op_t)255) {
pythontech 0:5868e8752d44 521 return false;
pythontech 0:5868e8752d44 522 }
pythontech 0:5868e8752d44 523 int rhs_sign = mp_obj_int_sign(arg1);
pythontech 0:5868e8752d44 524 if (op <= MP_BINARY_OP_RSHIFT) {
pythontech 0:5868e8752d44 525 // << and >> can't have negative rhs
pythontech 0:5868e8752d44 526 if (rhs_sign < 0) {
pythontech 0:5868e8752d44 527 return false;
pythontech 0:5868e8752d44 528 }
pythontech 0:5868e8752d44 529 } else if (op >= MP_BINARY_OP_FLOOR_DIVIDE) {
pythontech 0:5868e8752d44 530 // % and // can't have zero rhs
pythontech 0:5868e8752d44 531 if (rhs_sign == 0) {
pythontech 0:5868e8752d44 532 return false;
pythontech 0:5868e8752d44 533 }
pythontech 0:5868e8752d44 534 }
pythontech 0:5868e8752d44 535 arg0 = mp_binary_op(op, arg0, arg1);
pythontech 0:5868e8752d44 536 }
pythontech 0:5868e8752d44 537 } else if (rule->rule_id == RULE_factor_2) {
pythontech 0:5868e8752d44 538 // folding for unary ops: + - ~
pythontech 0:5868e8752d44 539 mp_parse_node_t pn = peek_result(parser, 0);
pythontech 0:5868e8752d44 540 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 541 return false;
pythontech 0:5868e8752d44 542 }
pythontech 0:5868e8752d44 543 mp_token_kind_t tok = MP_PARSE_NODE_LEAF_ARG(peek_result(parser, 1));
pythontech 0:5868e8752d44 544 mp_unary_op_t op;
pythontech 0:5868e8752d44 545 if (tok == MP_TOKEN_OP_PLUS) {
pythontech 0:5868e8752d44 546 op = MP_UNARY_OP_POSITIVE;
pythontech 0:5868e8752d44 547 } else if (tok == MP_TOKEN_OP_MINUS) {
pythontech 0:5868e8752d44 548 op = MP_UNARY_OP_NEGATIVE;
pythontech 0:5868e8752d44 549 } else {
pythontech 0:5868e8752d44 550 assert(tok == MP_TOKEN_OP_TILDE); // should be
pythontech 0:5868e8752d44 551 op = MP_UNARY_OP_INVERT;
pythontech 0:5868e8752d44 552 }
pythontech 0:5868e8752d44 553 arg0 = mp_unary_op(op, arg0);
pythontech 0:5868e8752d44 554
pythontech 0:5868e8752d44 555 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 556 } else if (rule->rule_id == RULE_expr_stmt) {
pythontech 0:5868e8752d44 557 mp_parse_node_t pn1 = peek_result(parser, 0);
pythontech 0:5868e8752d44 558 if (!MP_PARSE_NODE_IS_NULL(pn1)
pythontech 0:5868e8752d44 559 && !(MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_expr_stmt_augassign)
pythontech 0:5868e8752d44 560 || MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_expr_stmt_assign_list))) {
pythontech 0:5868e8752d44 561 // this node is of the form <x> = <y>
pythontech 0:5868e8752d44 562 mp_parse_node_t pn0 = peek_result(parser, 1);
pythontech 0:5868e8752d44 563 if (MP_PARSE_NODE_IS_ID(pn0)
Colin Hogben 2:c89e95946844 564 && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_atom_expr_normal)
pythontech 0:5868e8752d44 565 && MP_PARSE_NODE_IS_ID(((mp_parse_node_struct_t*)pn1)->nodes[0])
pythontech 0:5868e8752d44 566 && MP_PARSE_NODE_LEAF_ARG(((mp_parse_node_struct_t*)pn1)->nodes[0]) == MP_QSTR_const
pythontech 0:5868e8752d44 567 && MP_PARSE_NODE_IS_STRUCT_KIND(((mp_parse_node_struct_t*)pn1)->nodes[1], RULE_trailer_paren)
pythontech 0:5868e8752d44 568 ) {
pythontech 0:5868e8752d44 569 // code to assign dynamic constants: id = const(value)
pythontech 0:5868e8752d44 570
pythontech 0:5868e8752d44 571 // get the id
pythontech 0:5868e8752d44 572 qstr id = MP_PARSE_NODE_LEAF_ARG(pn0);
pythontech 0:5868e8752d44 573
pythontech 0:5868e8752d44 574 // get the value
pythontech 0:5868e8752d44 575 mp_parse_node_t pn_value = ((mp_parse_node_struct_t*)((mp_parse_node_struct_t*)pn1)->nodes[1])->nodes[0];
pythontech 0:5868e8752d44 576 if (!MP_PARSE_NODE_IS_SMALL_INT(pn_value)) {
pythontech 0:5868e8752d44 577 parser->parse_error = PARSE_ERROR_CONST;
pythontech 0:5868e8752d44 578 return false;
pythontech 0:5868e8752d44 579 }
pythontech 0:5868e8752d44 580 mp_int_t value = MP_PARSE_NODE_LEAF_SMALL_INT(pn_value);
pythontech 0:5868e8752d44 581
pythontech 0:5868e8752d44 582 // store the value in the table of dynamic constants
pythontech 0:5868e8752d44 583 mp_map_elem_t *elem = mp_map_lookup(&parser->consts, MP_OBJ_NEW_QSTR(id), MP_MAP_LOOKUP_ADD_IF_NOT_FOUND);
pythontech 0:5868e8752d44 584 assert(elem->value == MP_OBJ_NULL);
pythontech 0:5868e8752d44 585 elem->value = MP_OBJ_NEW_SMALL_INT(value);
pythontech 0:5868e8752d44 586
pythontech 0:5868e8752d44 587 // replace const(value) with value
pythontech 0:5868e8752d44 588 pop_result(parser);
pythontech 0:5868e8752d44 589 push_result_node(parser, pn_value);
pythontech 0:5868e8752d44 590
pythontech 0:5868e8752d44 591 // finished folding this assignment, but we still want it to be part of the tree
pythontech 0:5868e8752d44 592 return false;
pythontech 0:5868e8752d44 593 }
pythontech 0:5868e8752d44 594 }
pythontech 0:5868e8752d44 595 return false;
pythontech 0:5868e8752d44 596 #endif
pythontech 0:5868e8752d44 597
pythontech 0:5868e8752d44 598 #if MICROPY_COMP_MODULE_CONST
Colin Hogben 2:c89e95946844 599 } else if (rule->rule_id == RULE_atom_expr_normal) {
Colin Hogben 2:c89e95946844 600 mp_parse_node_t pn0 = peek_result(parser, 1);
Colin Hogben 2:c89e95946844 601 mp_parse_node_t pn1 = peek_result(parser, 0);
pythontech 0:5868e8752d44 602 if (!(MP_PARSE_NODE_IS_ID(pn0)
Colin Hogben 2:c89e95946844 603 && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_trailer_period))) {
pythontech 0:5868e8752d44 604 return false;
pythontech 0:5868e8752d44 605 }
pythontech 0:5868e8752d44 606 // id1.id2
pythontech 0:5868e8752d44 607 // look it up in constant table, see if it can be replaced with an integer
pythontech 0:5868e8752d44 608 mp_parse_node_struct_t *pns1 = (mp_parse_node_struct_t*)pn1;
pythontech 0:5868e8752d44 609 assert(MP_PARSE_NODE_IS_ID(pns1->nodes[0]));
pythontech 0:5868e8752d44 610 qstr q_base = MP_PARSE_NODE_LEAF_ARG(pn0);
pythontech 0:5868e8752d44 611 qstr q_attr = MP_PARSE_NODE_LEAF_ARG(pns1->nodes[0]);
pythontech 0:5868e8752d44 612 mp_map_elem_t *elem = mp_map_lookup((mp_map_t*)&mp_constants_map, MP_OBJ_NEW_QSTR(q_base), MP_MAP_LOOKUP);
pythontech 0:5868e8752d44 613 if (elem == NULL) {
pythontech 0:5868e8752d44 614 return false;
pythontech 0:5868e8752d44 615 }
pythontech 0:5868e8752d44 616 mp_obj_t dest[2];
pythontech 0:5868e8752d44 617 mp_load_method_maybe(elem->value, q_attr, dest);
pythontech 0:5868e8752d44 618 if (!(dest[0] != MP_OBJ_NULL && MP_OBJ_IS_INT(dest[0]) && dest[1] == MP_OBJ_NULL)) {
pythontech 0:5868e8752d44 619 return false;
pythontech 0:5868e8752d44 620 }
pythontech 0:5868e8752d44 621 arg0 = dest[0];
pythontech 0:5868e8752d44 622 #endif
pythontech 0:5868e8752d44 623
pythontech 0:5868e8752d44 624 } else {
pythontech 0:5868e8752d44 625 return false;
pythontech 0:5868e8752d44 626 }
pythontech 0:5868e8752d44 627
pythontech 0:5868e8752d44 628 // success folding this rule
pythontech 0:5868e8752d44 629
pythontech 0:5868e8752d44 630 for (size_t i = num_args; i > 0; i--) {
pythontech 0:5868e8752d44 631 pop_result(parser);
pythontech 0:5868e8752d44 632 }
pythontech 0:5868e8752d44 633 if (MP_OBJ_IS_SMALL_INT(arg0)) {
pythontech 0:5868e8752d44 634 push_result_node(parser, mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(arg0)));
pythontech 0:5868e8752d44 635 } else {
pythontech 0:5868e8752d44 636 // TODO reuse memory for parse node struct?
pythontech 0:5868e8752d44 637 push_result_node(parser, make_node_const_object(parser, 0, arg0));
pythontech 0:5868e8752d44 638 }
pythontech 0:5868e8752d44 639
pythontech 0:5868e8752d44 640 return true;
pythontech 0:5868e8752d44 641 }
pythontech 0:5868e8752d44 642 #endif
pythontech 0:5868e8752d44 643
pythontech 0:5868e8752d44 644 STATIC void push_result_rule(parser_t *parser, size_t src_line, const rule_t *rule, size_t num_args) {
pythontech 0:5868e8752d44 645 // optimise away parenthesis around an expression if possible
pythontech 0:5868e8752d44 646 if (rule->rule_id == RULE_atom_paren) {
pythontech 0:5868e8752d44 647 // there should be just 1 arg for this rule
pythontech 0:5868e8752d44 648 mp_parse_node_t pn = peek_result(parser, 0);
pythontech 0:5868e8752d44 649 if (MP_PARSE_NODE_IS_NULL(pn)) {
pythontech 0:5868e8752d44 650 // need to keep parenthesis for ()
pythontech 0:5868e8752d44 651 } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, RULE_testlist_comp)) {
pythontech 0:5868e8752d44 652 // need to keep parenthesis for (a, b, ...)
pythontech 0:5868e8752d44 653 } else {
pythontech 0:5868e8752d44 654 // parenthesis around a single expression, so it's just the expression
pythontech 0:5868e8752d44 655 return;
pythontech 0:5868e8752d44 656 }
pythontech 0:5868e8752d44 657 }
pythontech 0:5868e8752d44 658
pythontech 0:5868e8752d44 659 #if MICROPY_COMP_CONST_FOLDING
pythontech 0:5868e8752d44 660 if (fold_constants(parser, rule, num_args)) {
pythontech 0:5868e8752d44 661 // we folded this rule so return straight away
pythontech 0:5868e8752d44 662 return;
pythontech 0:5868e8752d44 663 }
pythontech 0:5868e8752d44 664 #endif
pythontech 0:5868e8752d44 665
pythontech 0:5868e8752d44 666 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_parse_node_t) * num_args);
pythontech 0:5868e8752d44 667 if (pn == NULL) {
pythontech 0:5868e8752d44 668 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 669 return;
pythontech 0:5868e8752d44 670 }
pythontech 0:5868e8752d44 671 pn->source_line = src_line;
pythontech 0:5868e8752d44 672 pn->kind_num_nodes = (rule->rule_id & 0xff) | (num_args << 8);
pythontech 0:5868e8752d44 673 for (size_t i = num_args; i > 0; i--) {
pythontech 0:5868e8752d44 674 pn->nodes[i - 1] = pop_result(parser);
pythontech 0:5868e8752d44 675 }
pythontech 0:5868e8752d44 676 push_result_node(parser, (mp_parse_node_t)pn);
pythontech 0:5868e8752d44 677 }
pythontech 0:5868e8752d44 678
pythontech 0:5868e8752d44 679 mp_parse_tree_t mp_parse(mp_lexer_t *lex, mp_parse_input_kind_t input_kind) {
pythontech 0:5868e8752d44 680
pythontech 0:5868e8752d44 681 // initialise parser and allocate memory for its stacks
pythontech 0:5868e8752d44 682
pythontech 0:5868e8752d44 683 parser_t parser;
pythontech 0:5868e8752d44 684
pythontech 0:5868e8752d44 685 parser.parse_error = PARSE_ERROR_NONE;
pythontech 0:5868e8752d44 686
pythontech 0:5868e8752d44 687 parser.rule_stack_alloc = MICROPY_ALLOC_PARSE_RULE_INIT;
pythontech 0:5868e8752d44 688 parser.rule_stack_top = 0;
pythontech 0:5868e8752d44 689 parser.rule_stack = m_new_maybe(rule_stack_t, parser.rule_stack_alloc);
pythontech 0:5868e8752d44 690
pythontech 0:5868e8752d44 691 parser.result_stack_alloc = MICROPY_ALLOC_PARSE_RESULT_INIT;
pythontech 0:5868e8752d44 692 parser.result_stack_top = 0;
pythontech 0:5868e8752d44 693 parser.result_stack = m_new_maybe(mp_parse_node_t, parser.result_stack_alloc);
pythontech 0:5868e8752d44 694
pythontech 0:5868e8752d44 695 parser.lexer = lex;
pythontech 0:5868e8752d44 696
pythontech 0:5868e8752d44 697 parser.tree.chunk = NULL;
pythontech 0:5868e8752d44 698 parser.cur_chunk = NULL;
pythontech 0:5868e8752d44 699
pythontech 0:5868e8752d44 700 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 701 mp_map_init(&parser.consts, 0);
pythontech 0:5868e8752d44 702 #endif
pythontech 0:5868e8752d44 703
pythontech 0:5868e8752d44 704 // check if we could allocate the stacks
pythontech 0:5868e8752d44 705 if (parser.rule_stack == NULL || parser.result_stack == NULL) {
pythontech 0:5868e8752d44 706 goto memory_error;
pythontech 0:5868e8752d44 707 }
pythontech 0:5868e8752d44 708
pythontech 0:5868e8752d44 709 // work out the top-level rule to use, and push it on the stack
pythontech 0:5868e8752d44 710 size_t top_level_rule;
pythontech 0:5868e8752d44 711 switch (input_kind) {
pythontech 0:5868e8752d44 712 case MP_PARSE_SINGLE_INPUT: top_level_rule = RULE_single_input; break;
pythontech 0:5868e8752d44 713 case MP_PARSE_EVAL_INPUT: top_level_rule = RULE_eval_input; break;
pythontech 0:5868e8752d44 714 default: top_level_rule = RULE_file_input;
pythontech 0:5868e8752d44 715 }
pythontech 0:5868e8752d44 716 push_rule(&parser, lex->tok_line, rules[top_level_rule], 0);
pythontech 0:5868e8752d44 717
pythontech 0:5868e8752d44 718 // parse!
pythontech 0:5868e8752d44 719
pythontech 0:5868e8752d44 720 size_t n, i; // state for the current rule
pythontech 0:5868e8752d44 721 size_t rule_src_line; // source line for the first token matched by the current rule
pythontech 0:5868e8752d44 722 bool backtrack = false;
pythontech 0:5868e8752d44 723 const rule_t *rule = NULL;
pythontech 0:5868e8752d44 724
pythontech 0:5868e8752d44 725 for (;;) {
pythontech 0:5868e8752d44 726 next_rule:
pythontech 0:5868e8752d44 727 if (parser.rule_stack_top == 0 || parser.parse_error) {
pythontech 0:5868e8752d44 728 break;
pythontech 0:5868e8752d44 729 }
pythontech 0:5868e8752d44 730
pythontech 0:5868e8752d44 731 pop_rule(&parser, &rule, &i, &rule_src_line);
pythontech 0:5868e8752d44 732 n = rule->act & RULE_ACT_ARG_MASK;
pythontech 0:5868e8752d44 733
pythontech 0:5868e8752d44 734 /*
pythontech 0:5868e8752d44 735 // debugging
pythontech 0:5868e8752d44 736 printf("depth=%d ", parser.rule_stack_top);
pythontech 0:5868e8752d44 737 for (int j = 0; j < parser.rule_stack_top; ++j) {
pythontech 0:5868e8752d44 738 printf(" ");
pythontech 0:5868e8752d44 739 }
pythontech 0:5868e8752d44 740 printf("%s n=%d i=%d bt=%d\n", rule->rule_name, n, i, backtrack);
pythontech 0:5868e8752d44 741 */
pythontech 0:5868e8752d44 742
pythontech 0:5868e8752d44 743 switch (rule->act & RULE_ACT_KIND_MASK) {
pythontech 0:5868e8752d44 744 case RULE_ACT_OR:
pythontech 0:5868e8752d44 745 if (i > 0 && !backtrack) {
pythontech 0:5868e8752d44 746 goto next_rule;
pythontech 0:5868e8752d44 747 } else {
pythontech 0:5868e8752d44 748 backtrack = false;
pythontech 0:5868e8752d44 749 }
pythontech 0:5868e8752d44 750 for (; i < n; ++i) {
pythontech 0:5868e8752d44 751 uint16_t kind = rule->arg[i] & RULE_ARG_KIND_MASK;
pythontech 0:5868e8752d44 752 if (kind == RULE_ARG_TOK) {
pythontech 0:5868e8752d44 753 if (lex->tok_kind == (rule->arg[i] & RULE_ARG_ARG_MASK)) {
pythontech 0:5868e8752d44 754 push_result_token(&parser);
pythontech 0:5868e8752d44 755 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 756 goto next_rule;
pythontech 0:5868e8752d44 757 }
pythontech 0:5868e8752d44 758 } else {
pythontech 0:5868e8752d44 759 assert(kind == RULE_ARG_RULE);
pythontech 0:5868e8752d44 760 if (i + 1 < n) {
pythontech 0:5868e8752d44 761 push_rule(&parser, rule_src_line, rule, i + 1); // save this or-rule
pythontech 0:5868e8752d44 762 }
pythontech 0:5868e8752d44 763 push_rule_from_arg(&parser, rule->arg[i]); // push child of or-rule
pythontech 0:5868e8752d44 764 goto next_rule;
pythontech 0:5868e8752d44 765 }
pythontech 0:5868e8752d44 766 }
pythontech 0:5868e8752d44 767 backtrack = true;
pythontech 0:5868e8752d44 768 break;
pythontech 0:5868e8752d44 769
pythontech 0:5868e8752d44 770 case RULE_ACT_AND: {
pythontech 0:5868e8752d44 771
pythontech 0:5868e8752d44 772 // failed, backtrack if we can, else syntax error
pythontech 0:5868e8752d44 773 if (backtrack) {
pythontech 0:5868e8752d44 774 assert(i > 0);
pythontech 0:5868e8752d44 775 if ((rule->arg[i - 1] & RULE_ARG_KIND_MASK) == RULE_ARG_OPT_RULE) {
pythontech 0:5868e8752d44 776 // an optional rule that failed, so continue with next arg
pythontech 0:5868e8752d44 777 push_result_node(&parser, MP_PARSE_NODE_NULL);
pythontech 0:5868e8752d44 778 backtrack = false;
pythontech 0:5868e8752d44 779 } else {
pythontech 0:5868e8752d44 780 // a mandatory rule that failed, so propagate backtrack
pythontech 0:5868e8752d44 781 if (i > 1) {
pythontech 0:5868e8752d44 782 // already eaten tokens so can't backtrack
pythontech 0:5868e8752d44 783 goto syntax_error;
pythontech 0:5868e8752d44 784 } else {
pythontech 0:5868e8752d44 785 goto next_rule;
pythontech 0:5868e8752d44 786 }
pythontech 0:5868e8752d44 787 }
pythontech 0:5868e8752d44 788 }
pythontech 0:5868e8752d44 789
pythontech 0:5868e8752d44 790 // progress through the rule
pythontech 0:5868e8752d44 791 for (; i < n; ++i) {
pythontech 0:5868e8752d44 792 switch (rule->arg[i] & RULE_ARG_KIND_MASK) {
pythontech 0:5868e8752d44 793 case RULE_ARG_TOK: {
pythontech 0:5868e8752d44 794 // need to match a token
pythontech 0:5868e8752d44 795 mp_token_kind_t tok_kind = rule->arg[i] & RULE_ARG_ARG_MASK;
pythontech 0:5868e8752d44 796 if (lex->tok_kind == tok_kind) {
pythontech 0:5868e8752d44 797 // matched token
pythontech 0:5868e8752d44 798 if (tok_kind == MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 799 push_result_token(&parser);
pythontech 0:5868e8752d44 800 }
pythontech 0:5868e8752d44 801 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 802 } else {
pythontech 0:5868e8752d44 803 // failed to match token
pythontech 0:5868e8752d44 804 if (i > 0) {
pythontech 0:5868e8752d44 805 // already eaten tokens so can't backtrack
pythontech 0:5868e8752d44 806 goto syntax_error;
pythontech 0:5868e8752d44 807 } else {
pythontech 0:5868e8752d44 808 // this rule failed, so backtrack
pythontech 0:5868e8752d44 809 backtrack = true;
pythontech 0:5868e8752d44 810 goto next_rule;
pythontech 0:5868e8752d44 811 }
pythontech 0:5868e8752d44 812 }
pythontech 0:5868e8752d44 813 break;
pythontech 0:5868e8752d44 814 }
pythontech 0:5868e8752d44 815 case RULE_ARG_RULE:
pythontech 0:5868e8752d44 816 case RULE_ARG_OPT_RULE:
pythontech 0:5868e8752d44 817 rule_and_no_other_choice:
pythontech 0:5868e8752d44 818 push_rule(&parser, rule_src_line, rule, i + 1); // save this and-rule
pythontech 0:5868e8752d44 819 push_rule_from_arg(&parser, rule->arg[i]); // push child of and-rule
pythontech 0:5868e8752d44 820 goto next_rule;
pythontech 0:5868e8752d44 821 default:
pythontech 0:5868e8752d44 822 assert(0);
pythontech 0:5868e8752d44 823 goto rule_and_no_other_choice; // to help flow control analysis
pythontech 0:5868e8752d44 824 }
pythontech 0:5868e8752d44 825 }
pythontech 0:5868e8752d44 826
pythontech 0:5868e8752d44 827 assert(i == n);
pythontech 0:5868e8752d44 828
pythontech 0:5868e8752d44 829 // matched the rule, so now build the corresponding parse_node
pythontech 0:5868e8752d44 830
pythontech 0:5868e8752d44 831 #if !MICROPY_ENABLE_DOC_STRING
pythontech 0:5868e8752d44 832 // this code discards lonely statements, such as doc strings
pythontech 0:5868e8752d44 833 if (input_kind != MP_PARSE_SINGLE_INPUT && rule->rule_id == RULE_expr_stmt && peek_result(&parser, 0) == MP_PARSE_NODE_NULL) {
pythontech 0:5868e8752d44 834 mp_parse_node_t p = peek_result(&parser, 1);
pythontech 0:5868e8752d44 835 if ((MP_PARSE_NODE_IS_LEAF(p) && !MP_PARSE_NODE_IS_ID(p)) || MP_PARSE_NODE_IS_STRUCT_KIND(p, RULE_string)) {
pythontech 0:5868e8752d44 836 pop_result(&parser); // MP_PARSE_NODE_NULL
pythontech 0:5868e8752d44 837 mp_parse_node_t pn = pop_result(&parser); // possibly RULE_string
pythontech 0:5868e8752d44 838 if (MP_PARSE_NODE_IS_STRUCT(pn)) {
pythontech 0:5868e8752d44 839 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t *)pn;
pythontech 0:5868e8752d44 840 if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_string) {
pythontech 0:5868e8752d44 841 m_del(char, (char*)pns->nodes[0], (size_t)pns->nodes[1]);
pythontech 0:5868e8752d44 842 }
pythontech 0:5868e8752d44 843 }
pythontech 0:5868e8752d44 844 push_result_rule(&parser, rule_src_line, rules[RULE_pass_stmt], 0);
pythontech 0:5868e8752d44 845 break;
pythontech 0:5868e8752d44 846 }
pythontech 0:5868e8752d44 847 }
pythontech 0:5868e8752d44 848 #endif
pythontech 0:5868e8752d44 849
Colin Hogben 2:c89e95946844 850 // count number of arguments for the parse node
Colin Hogben 2:c89e95946844 851 i = 0;
Colin Hogben 2:c89e95946844 852 size_t num_not_nil = 0;
Colin Hogben 2:c89e95946844 853 for (size_t x = n; x > 0;) {
Colin Hogben 2:c89e95946844 854 --x;
Colin Hogben 2:c89e95946844 855 if ((rule->arg[x] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) {
Colin Hogben 2:c89e95946844 856 mp_token_kind_t tok_kind = rule->arg[x] & RULE_ARG_ARG_MASK;
Colin Hogben 2:c89e95946844 857 if (tok_kind == MP_TOKEN_NAME) {
Colin Hogben 2:c89e95946844 858 // only tokens which were names are pushed to stack
Colin Hogben 2:c89e95946844 859 i += 1;
Colin Hogben 2:c89e95946844 860 num_not_nil += 1;
Colin Hogben 2:c89e95946844 861 }
Colin Hogben 2:c89e95946844 862 } else {
Colin Hogben 2:c89e95946844 863 // rules are always pushed
Colin Hogben 2:c89e95946844 864 if (peek_result(&parser, i) != MP_PARSE_NODE_NULL) {
Colin Hogben 2:c89e95946844 865 num_not_nil += 1;
Colin Hogben 2:c89e95946844 866 }
Colin Hogben 2:c89e95946844 867 i += 1;
Colin Hogben 2:c89e95946844 868 }
pythontech 0:5868e8752d44 869 }
pythontech 0:5868e8752d44 870
Colin Hogben 2:c89e95946844 871 if (num_not_nil == 1 && (rule->act & RULE_ACT_ALLOW_IDENT)) {
Colin Hogben 2:c89e95946844 872 // this rule has only 1 argument and should not be emitted
pythontech 0:5868e8752d44 873 mp_parse_node_t pn = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 874 for (size_t x = 0; x < i; ++x) {
pythontech 0:5868e8752d44 875 mp_parse_node_t pn2 = pop_result(&parser);
pythontech 0:5868e8752d44 876 if (pn2 != MP_PARSE_NODE_NULL) {
pythontech 0:5868e8752d44 877 pn = pn2;
pythontech 0:5868e8752d44 878 }
pythontech 0:5868e8752d44 879 }
pythontech 0:5868e8752d44 880 push_result_node(&parser, pn);
Colin Hogben 2:c89e95946844 881 } else {
Colin Hogben 2:c89e95946844 882 // this rule must be emitted
Colin Hogben 2:c89e95946844 883
Colin Hogben 2:c89e95946844 884 if (rule->act & RULE_ACT_ADD_BLANK) {
Colin Hogben 2:c89e95946844 885 // and add an extra blank node at the end (used by the compiler to store data)
Colin Hogben 2:c89e95946844 886 push_result_node(&parser, MP_PARSE_NODE_NULL);
Colin Hogben 2:c89e95946844 887 i += 1;
Colin Hogben 2:c89e95946844 888 }
Colin Hogben 2:c89e95946844 889
Colin Hogben 2:c89e95946844 890 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 891 }
pythontech 0:5868e8752d44 892 break;
pythontech 0:5868e8752d44 893 }
pythontech 0:5868e8752d44 894
pythontech 0:5868e8752d44 895 case RULE_ACT_LIST: {
pythontech 0:5868e8752d44 896 // n=2 is: item item*
pythontech 0:5868e8752d44 897 // n=1 is: item (sep item)*
pythontech 0:5868e8752d44 898 // n=3 is: item (sep item)* [sep]
pythontech 0:5868e8752d44 899 bool had_trailing_sep;
pythontech 0:5868e8752d44 900 if (backtrack) {
pythontech 0:5868e8752d44 901 list_backtrack:
pythontech 0:5868e8752d44 902 had_trailing_sep = false;
pythontech 0:5868e8752d44 903 if (n == 2) {
pythontech 0:5868e8752d44 904 if (i == 1) {
pythontech 0:5868e8752d44 905 // fail on item, first time round; propagate backtrack
pythontech 0:5868e8752d44 906 goto next_rule;
pythontech 0:5868e8752d44 907 } else {
pythontech 0:5868e8752d44 908 // fail on item, in later rounds; finish with this rule
pythontech 0:5868e8752d44 909 backtrack = false;
pythontech 0:5868e8752d44 910 }
pythontech 0:5868e8752d44 911 } else {
pythontech 0:5868e8752d44 912 if (i == 1) {
pythontech 0:5868e8752d44 913 // fail on item, first time round; propagate backtrack
pythontech 0:5868e8752d44 914 goto next_rule;
pythontech 0:5868e8752d44 915 } else if ((i & 1) == 1) {
pythontech 0:5868e8752d44 916 // fail on item, in later rounds; have eaten tokens so can't backtrack
pythontech 0:5868e8752d44 917 if (n == 3) {
pythontech 0:5868e8752d44 918 // list allows trailing separator; finish parsing list
pythontech 0:5868e8752d44 919 had_trailing_sep = true;
pythontech 0:5868e8752d44 920 backtrack = false;
pythontech 0:5868e8752d44 921 } else {
pythontech 0:5868e8752d44 922 // list doesn't allowing trailing separator; fail
pythontech 0:5868e8752d44 923 goto syntax_error;
pythontech 0:5868e8752d44 924 }
pythontech 0:5868e8752d44 925 } else {
pythontech 0:5868e8752d44 926 // fail on separator; finish parsing list
pythontech 0:5868e8752d44 927 backtrack = false;
pythontech 0:5868e8752d44 928 }
pythontech 0:5868e8752d44 929 }
pythontech 0:5868e8752d44 930 } else {
pythontech 0:5868e8752d44 931 for (;;) {
pythontech 0:5868e8752d44 932 size_t arg = rule->arg[i & 1 & n];
pythontech 0:5868e8752d44 933 switch (arg & RULE_ARG_KIND_MASK) {
pythontech 0:5868e8752d44 934 case RULE_ARG_TOK:
pythontech 0:5868e8752d44 935 if (lex->tok_kind == (arg & RULE_ARG_ARG_MASK)) {
pythontech 0:5868e8752d44 936 if (i & 1 & n) {
pythontech 0:5868e8752d44 937 // separators which are tokens are not pushed to result stack
pythontech 0:5868e8752d44 938 } else {
pythontech 0:5868e8752d44 939 push_result_token(&parser);
pythontech 0:5868e8752d44 940 }
pythontech 0:5868e8752d44 941 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 942 // got element of list, so continue parsing list
pythontech 0:5868e8752d44 943 i += 1;
pythontech 0:5868e8752d44 944 } else {
pythontech 0:5868e8752d44 945 // couldn't get element of list
pythontech 0:5868e8752d44 946 i += 1;
pythontech 0:5868e8752d44 947 backtrack = true;
pythontech 0:5868e8752d44 948 goto list_backtrack;
pythontech 0:5868e8752d44 949 }
pythontech 0:5868e8752d44 950 break;
pythontech 0:5868e8752d44 951 case RULE_ARG_RULE:
pythontech 0:5868e8752d44 952 rule_list_no_other_choice:
pythontech 0:5868e8752d44 953 push_rule(&parser, rule_src_line, rule, i + 1); // save this list-rule
pythontech 0:5868e8752d44 954 push_rule_from_arg(&parser, arg); // push child of list-rule
pythontech 0:5868e8752d44 955 goto next_rule;
pythontech 0:5868e8752d44 956 default:
pythontech 0:5868e8752d44 957 assert(0);
pythontech 0:5868e8752d44 958 goto rule_list_no_other_choice; // to help flow control analysis
pythontech 0:5868e8752d44 959 }
pythontech 0:5868e8752d44 960 }
pythontech 0:5868e8752d44 961 }
pythontech 0:5868e8752d44 962 assert(i >= 1);
pythontech 0:5868e8752d44 963
pythontech 0:5868e8752d44 964 // compute number of elements in list, result in i
pythontech 0:5868e8752d44 965 i -= 1;
pythontech 0:5868e8752d44 966 if ((n & 1) && (rule->arg[1] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) {
pythontech 0:5868e8752d44 967 // don't count separators when they are tokens
pythontech 0:5868e8752d44 968 i = (i + 1) / 2;
pythontech 0:5868e8752d44 969 }
pythontech 0:5868e8752d44 970
pythontech 0:5868e8752d44 971 if (i == 1) {
pythontech 0:5868e8752d44 972 // list matched single item
pythontech 0:5868e8752d44 973 if (had_trailing_sep) {
pythontech 0:5868e8752d44 974 // if there was a trailing separator, make a list of a single item
pythontech 0:5868e8752d44 975 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 976 } else {
pythontech 0:5868e8752d44 977 // just leave single item on stack (ie don't wrap in a list)
pythontech 0:5868e8752d44 978 }
pythontech 0:5868e8752d44 979 } else {
pythontech 0:5868e8752d44 980 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 981 }
pythontech 0:5868e8752d44 982 break;
pythontech 0:5868e8752d44 983 }
pythontech 0:5868e8752d44 984
pythontech 0:5868e8752d44 985 default:
pythontech 0:5868e8752d44 986 assert(0);
pythontech 0:5868e8752d44 987 }
pythontech 0:5868e8752d44 988 }
pythontech 0:5868e8752d44 989
pythontech 0:5868e8752d44 990 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 991 mp_map_deinit(&parser.consts);
pythontech 0:5868e8752d44 992 #endif
pythontech 0:5868e8752d44 993
pythontech 0:5868e8752d44 994 // truncate final chunk and link into chain of chunks
pythontech 0:5868e8752d44 995 if (parser.cur_chunk != NULL) {
pythontech 0:5868e8752d44 996 (void)m_renew(byte, parser.cur_chunk,
pythontech 0:5868e8752d44 997 sizeof(mp_parse_chunk_t) + parser.cur_chunk->alloc,
pythontech 0:5868e8752d44 998 sizeof(mp_parse_chunk_t) + parser.cur_chunk->union_.used);
pythontech 0:5868e8752d44 999 parser.cur_chunk->alloc = parser.cur_chunk->union_.used;
pythontech 0:5868e8752d44 1000 parser.cur_chunk->union_.next = parser.tree.chunk;
pythontech 0:5868e8752d44 1001 parser.tree.chunk = parser.cur_chunk;
pythontech 0:5868e8752d44 1002 }
pythontech 0:5868e8752d44 1003
pythontech 0:5868e8752d44 1004 mp_obj_t exc;
pythontech 0:5868e8752d44 1005
pythontech 0:5868e8752d44 1006 if (parser.parse_error) {
pythontech 0:5868e8752d44 1007 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 1008 if (parser.parse_error == PARSE_ERROR_CONST) {
pythontech 0:5868e8752d44 1009 exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
pythontech 0:5868e8752d44 1010 "constant must be an integer");
pythontech 0:5868e8752d44 1011 } else
pythontech 0:5868e8752d44 1012 #endif
pythontech 0:5868e8752d44 1013 {
pythontech 0:5868e8752d44 1014 assert(parser.parse_error == PARSE_ERROR_MEMORY);
pythontech 0:5868e8752d44 1015 memory_error:
pythontech 0:5868e8752d44 1016 exc = mp_obj_new_exception_msg(&mp_type_MemoryError,
pythontech 0:5868e8752d44 1017 "parser could not allocate enough memory");
pythontech 0:5868e8752d44 1018 }
pythontech 0:5868e8752d44 1019 parser.tree.root = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 1020 } else if (
pythontech 0:5868e8752d44 1021 lex->tok_kind != MP_TOKEN_END // check we are at the end of the token stream
pythontech 0:5868e8752d44 1022 || parser.result_stack_top == 0 // check that we got a node (can fail on empty input)
pythontech 0:5868e8752d44 1023 ) {
pythontech 0:5868e8752d44 1024 syntax_error:
pythontech 0:5868e8752d44 1025 if (lex->tok_kind == MP_TOKEN_INDENT) {
pythontech 0:5868e8752d44 1026 exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
pythontech 0:5868e8752d44 1027 "unexpected indent");
pythontech 0:5868e8752d44 1028 } else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
pythontech 0:5868e8752d44 1029 exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
pythontech 0:5868e8752d44 1030 "unindent does not match any outer indentation level");
pythontech 0:5868e8752d44 1031 } else {
pythontech 0:5868e8752d44 1032 exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
pythontech 0:5868e8752d44 1033 "invalid syntax");
pythontech 0:5868e8752d44 1034 }
pythontech 0:5868e8752d44 1035 parser.tree.root = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 1036 } else {
pythontech 0:5868e8752d44 1037 // no errors
pythontech 0:5868e8752d44 1038
pythontech 0:5868e8752d44 1039 //result_stack_show(parser);
pythontech 0:5868e8752d44 1040 //printf("rule stack alloc: %d\n", parser.rule_stack_alloc);
pythontech 0:5868e8752d44 1041 //printf("result stack alloc: %d\n", parser.result_stack_alloc);
pythontech 0:5868e8752d44 1042 //printf("number of parse nodes allocated: %d\n", num_parse_nodes_allocated);
pythontech 0:5868e8752d44 1043
pythontech 0:5868e8752d44 1044 // get the root parse node that we created
pythontech 0:5868e8752d44 1045 assert(parser.result_stack_top == 1);
pythontech 0:5868e8752d44 1046 exc = MP_OBJ_NULL;
pythontech 0:5868e8752d44 1047 parser.tree.root = parser.result_stack[0];
pythontech 0:5868e8752d44 1048 }
pythontech 0:5868e8752d44 1049
pythontech 0:5868e8752d44 1050 // free the memory that we don't need anymore
pythontech 0:5868e8752d44 1051 m_del(rule_stack_t, parser.rule_stack, parser.rule_stack_alloc);
pythontech 0:5868e8752d44 1052 m_del(mp_parse_node_t, parser.result_stack, parser.result_stack_alloc);
pythontech 0:5868e8752d44 1053 // we also free the lexer on behalf of the caller (see below)
pythontech 0:5868e8752d44 1054
pythontech 0:5868e8752d44 1055 if (exc != MP_OBJ_NULL) {
pythontech 0:5868e8752d44 1056 // had an error so raise the exception
pythontech 0:5868e8752d44 1057 // add traceback to give info about file name and location
pythontech 0:5868e8752d44 1058 // we don't have a 'block' name, so just pass the NULL qstr to indicate this
pythontech 0:5868e8752d44 1059 mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
pythontech 0:5868e8752d44 1060 mp_lexer_free(lex);
pythontech 0:5868e8752d44 1061 nlr_raise(exc);
pythontech 0:5868e8752d44 1062 } else {
pythontech 0:5868e8752d44 1063 mp_lexer_free(lex);
pythontech 0:5868e8752d44 1064 return parser.tree;
pythontech 0:5868e8752d44 1065 }
pythontech 0:5868e8752d44 1066 }
pythontech 0:5868e8752d44 1067
pythontech 0:5868e8752d44 1068 void mp_parse_tree_clear(mp_parse_tree_t *tree) {
pythontech 0:5868e8752d44 1069 mp_parse_chunk_t *chunk = tree->chunk;
pythontech 0:5868e8752d44 1070 while (chunk != NULL) {
pythontech 0:5868e8752d44 1071 mp_parse_chunk_t *next = chunk->union_.next;
pythontech 0:5868e8752d44 1072 m_del(byte, chunk, sizeof(mp_parse_chunk_t) + chunk->alloc);
pythontech 0:5868e8752d44 1073 chunk = next;
pythontech 0:5868e8752d44 1074 }
pythontech 0:5868e8752d44 1075 }
pythontech 0:5868e8752d44 1076
pythontech 0:5868e8752d44 1077 #endif // MICROPY_ENABLE_COMPILER