Port of MicroPython to the mbed platform. See micropython-repl for an interactive program.

Dependents:   micropython-repl

This a port of MicroPython to the mbed Classic platform.

This provides an interpreter running on the board's USB serial connection.

Getting Started

Import the micropython-repl program into your IDE workspace on developer.mbed.org. Compile and download to your board. Connect to the USB serial port in your usual manner. You should get a startup message similar to the following:

  MicroPython v1.7-155-gdddcdd8 on 2016-04-23; K64F with ARM
  Type "help()" for more information.
  >>>

Then you can start using micropython. For example:

  >>> from mbed import DigitalOut
  >>> from pins import LED1
  >>> led = DigitalOut(LED1)
  >>> led.write(1)

Requirements

You need approximately 100K of flash memory, so this will be no good for boards with smaller amounts of storage.

Caveats

This can be considered an alpha release of the port; things may not work; APIs may change in later releases. It is NOT an official part part the micropython project, so if anything doesn't work, blame me. If it does work, most of the credit is due to micropython.

  • Only a few of the mbed classes are available in micropython so far, and not all methods of those that are.
  • Only a few boards have their full range of pin names available; for others, only a few standard ones (USBTX, USBRX, LED1) are implemented.
  • The garbage collector is not yet implemented. The interpreter will gradually consume memory and then fail.
  • Exceptions from the mbed classes are not yet handled.
  • Asynchronous processing (e.g. events on inputs) is not supported.

Credits

  • Damien P. George and other contributors who created micropython.
  • Colin Hogben, author of this port.
Committer:
pythontech
Date:
Sat Apr 16 17:11:56 2016 +0000
Revision:
0:5868e8752d44
Child:
2:c89e95946844
Split off library from repl

Who changed what in which revision?

UserRevisionLine numberNew contents of line
pythontech 0:5868e8752d44 1 /*
pythontech 0:5868e8752d44 2 * This file is part of the Micro Python project, http://micropython.org/
pythontech 0:5868e8752d44 3 *
pythontech 0:5868e8752d44 4 * The MIT License (MIT)
pythontech 0:5868e8752d44 5 *
pythontech 0:5868e8752d44 6 * Copyright (c) 2013-2015 Damien P. George
pythontech 0:5868e8752d44 7 *
pythontech 0:5868e8752d44 8 * Permission is hereby granted, free of charge, to any person obtaining a copy
pythontech 0:5868e8752d44 9 * of this software and associated documentation files (the "Software"), to deal
pythontech 0:5868e8752d44 10 * in the Software without restriction, including without limitation the rights
pythontech 0:5868e8752d44 11 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
pythontech 0:5868e8752d44 12 * copies of the Software, and to permit persons to whom the Software is
pythontech 0:5868e8752d44 13 * furnished to do so, subject to the following conditions:
pythontech 0:5868e8752d44 14 *
pythontech 0:5868e8752d44 15 * The above copyright notice and this permission notice shall be included in
pythontech 0:5868e8752d44 16 * all copies or substantial portions of the Software.
pythontech 0:5868e8752d44 17 *
pythontech 0:5868e8752d44 18 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
pythontech 0:5868e8752d44 19 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
pythontech 0:5868e8752d44 20 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
pythontech 0:5868e8752d44 21 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
pythontech 0:5868e8752d44 22 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
pythontech 0:5868e8752d44 23 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
pythontech 0:5868e8752d44 24 * THE SOFTWARE.
pythontech 0:5868e8752d44 25 */
pythontech 0:5868e8752d44 26
pythontech 0:5868e8752d44 27 #include <stdbool.h>
pythontech 0:5868e8752d44 28 #include <stdint.h>
pythontech 0:5868e8752d44 29 #include <stdio.h>
pythontech 0:5868e8752d44 30 #include <unistd.h> // for ssize_t
pythontech 0:5868e8752d44 31 #include <assert.h>
pythontech 0:5868e8752d44 32 #include <string.h>
pythontech 0:5868e8752d44 33
pythontech 0:5868e8752d44 34 #include "py/nlr.h"
pythontech 0:5868e8752d44 35 #include "py/lexer.h"
pythontech 0:5868e8752d44 36 #include "py/parse.h"
pythontech 0:5868e8752d44 37 #include "py/parsenum.h"
pythontech 0:5868e8752d44 38 #include "py/runtime0.h"
pythontech 0:5868e8752d44 39 #include "py/runtime.h"
pythontech 0:5868e8752d44 40 #include "py/objint.h"
pythontech 0:5868e8752d44 41 #include "py/builtin.h"
pythontech 0:5868e8752d44 42
pythontech 0:5868e8752d44 43 #if MICROPY_ENABLE_COMPILER
pythontech 0:5868e8752d44 44
pythontech 0:5868e8752d44 45 #define RULE_ACT_ARG_MASK (0x0f)
pythontech 0:5868e8752d44 46 #define RULE_ACT_KIND_MASK (0x30)
pythontech 0:5868e8752d44 47 #define RULE_ACT_ALLOW_IDENT (0x40)
pythontech 0:5868e8752d44 48 #define RULE_ACT_ADD_BLANK (0x80)
pythontech 0:5868e8752d44 49 #define RULE_ACT_OR (0x10)
pythontech 0:5868e8752d44 50 #define RULE_ACT_AND (0x20)
pythontech 0:5868e8752d44 51 #define RULE_ACT_LIST (0x30)
pythontech 0:5868e8752d44 52
pythontech 0:5868e8752d44 53 #define RULE_ARG_KIND_MASK (0xf000)
pythontech 0:5868e8752d44 54 #define RULE_ARG_ARG_MASK (0x0fff)
pythontech 0:5868e8752d44 55 #define RULE_ARG_TOK (0x1000)
pythontech 0:5868e8752d44 56 #define RULE_ARG_RULE (0x2000)
pythontech 0:5868e8752d44 57 #define RULE_ARG_OPT_RULE (0x3000)
pythontech 0:5868e8752d44 58
pythontech 0:5868e8752d44 59 #define ADD_BLANK_NODE(rule) ((rule->act & RULE_ACT_ADD_BLANK) != 0)
pythontech 0:5868e8752d44 60
pythontech 0:5868e8752d44 61 // (un)comment to use rule names; for debugging
pythontech 0:5868e8752d44 62 //#define USE_RULE_NAME (1)
pythontech 0:5868e8752d44 63
pythontech 0:5868e8752d44 64 typedef struct _rule_t {
pythontech 0:5868e8752d44 65 byte rule_id;
pythontech 0:5868e8752d44 66 byte act;
pythontech 0:5868e8752d44 67 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 68 const char *rule_name;
pythontech 0:5868e8752d44 69 #endif
pythontech 0:5868e8752d44 70 uint16_t arg[];
pythontech 0:5868e8752d44 71 } rule_t;
pythontech 0:5868e8752d44 72
pythontech 0:5868e8752d44 73 enum {
pythontech 0:5868e8752d44 74 #define DEF_RULE(rule, comp, kind, ...) RULE_##rule,
pythontech 0:5868e8752d44 75 #include "py/grammar.h"
pythontech 0:5868e8752d44 76 #undef DEF_RULE
pythontech 0:5868e8752d44 77 RULE_maximum_number_of,
pythontech 0:5868e8752d44 78 RULE_string, // special node for non-interned string
pythontech 0:5868e8752d44 79 RULE_bytes, // special node for non-interned bytes
pythontech 0:5868e8752d44 80 RULE_const_object, // special node for a constant, generic Python object
pythontech 0:5868e8752d44 81 };
pythontech 0:5868e8752d44 82
pythontech 0:5868e8752d44 83 #define ident (RULE_ACT_ALLOW_IDENT)
pythontech 0:5868e8752d44 84 #define blank (RULE_ACT_ADD_BLANK)
pythontech 0:5868e8752d44 85 #define or(n) (RULE_ACT_OR | n)
pythontech 0:5868e8752d44 86 #define and(n) (RULE_ACT_AND | n)
pythontech 0:5868e8752d44 87 #define one_or_more (RULE_ACT_LIST | 2)
pythontech 0:5868e8752d44 88 #define list (RULE_ACT_LIST | 1)
pythontech 0:5868e8752d44 89 #define list_with_end (RULE_ACT_LIST | 3)
pythontech 0:5868e8752d44 90 #define tok(t) (RULE_ARG_TOK | MP_TOKEN_##t)
pythontech 0:5868e8752d44 91 #define rule(r) (RULE_ARG_RULE | RULE_##r)
pythontech 0:5868e8752d44 92 #define opt_rule(r) (RULE_ARG_OPT_RULE | RULE_##r)
pythontech 0:5868e8752d44 93 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 94 #define DEF_RULE(rule, comp, kind, ...) static const rule_t rule_##rule = { RULE_##rule, kind, #rule, { __VA_ARGS__ } };
pythontech 0:5868e8752d44 95 #else
pythontech 0:5868e8752d44 96 #define DEF_RULE(rule, comp, kind, ...) static const rule_t rule_##rule = { RULE_##rule, kind, { __VA_ARGS__ } };
pythontech 0:5868e8752d44 97 #endif
pythontech 0:5868e8752d44 98 #include "py/grammar.h"
pythontech 0:5868e8752d44 99 #undef or
pythontech 0:5868e8752d44 100 #undef and
pythontech 0:5868e8752d44 101 #undef list
pythontech 0:5868e8752d44 102 #undef list_with_end
pythontech 0:5868e8752d44 103 #undef tok
pythontech 0:5868e8752d44 104 #undef rule
pythontech 0:5868e8752d44 105 #undef opt_rule
pythontech 0:5868e8752d44 106 #undef one_or_more
pythontech 0:5868e8752d44 107 #undef DEF_RULE
pythontech 0:5868e8752d44 108
pythontech 0:5868e8752d44 109 STATIC const rule_t *rules[] = {
pythontech 0:5868e8752d44 110 #define DEF_RULE(rule, comp, kind, ...) &rule_##rule,
pythontech 0:5868e8752d44 111 #include "py/grammar.h"
pythontech 0:5868e8752d44 112 #undef DEF_RULE
pythontech 0:5868e8752d44 113 };
pythontech 0:5868e8752d44 114
pythontech 0:5868e8752d44 115 typedef struct _rule_stack_t {
pythontech 0:5868e8752d44 116 size_t src_line : 8 * sizeof(size_t) - 8; // maximum bits storing source line number
pythontech 0:5868e8752d44 117 size_t rule_id : 8; // this must be large enough to fit largest rule number
pythontech 0:5868e8752d44 118 size_t arg_i; // this dictates the maximum nodes in a "list" of things
pythontech 0:5868e8752d44 119 } rule_stack_t;
pythontech 0:5868e8752d44 120
pythontech 0:5868e8752d44 121 typedef struct _mp_parse_chunk_t {
pythontech 0:5868e8752d44 122 size_t alloc;
pythontech 0:5868e8752d44 123 union {
pythontech 0:5868e8752d44 124 size_t used;
pythontech 0:5868e8752d44 125 struct _mp_parse_chunk_t *next;
pythontech 0:5868e8752d44 126 } union_;
pythontech 0:5868e8752d44 127 byte data[];
pythontech 0:5868e8752d44 128 } mp_parse_chunk_t;
pythontech 0:5868e8752d44 129
pythontech 0:5868e8752d44 130 typedef enum {
pythontech 0:5868e8752d44 131 PARSE_ERROR_NONE = 0,
pythontech 0:5868e8752d44 132 PARSE_ERROR_MEMORY,
pythontech 0:5868e8752d44 133 PARSE_ERROR_CONST,
pythontech 0:5868e8752d44 134 } parse_error_t;
pythontech 0:5868e8752d44 135
pythontech 0:5868e8752d44 136 typedef struct _parser_t {
pythontech 0:5868e8752d44 137 parse_error_t parse_error;
pythontech 0:5868e8752d44 138
pythontech 0:5868e8752d44 139 size_t rule_stack_alloc;
pythontech 0:5868e8752d44 140 size_t rule_stack_top;
pythontech 0:5868e8752d44 141 rule_stack_t *rule_stack;
pythontech 0:5868e8752d44 142
pythontech 0:5868e8752d44 143 size_t result_stack_alloc;
pythontech 0:5868e8752d44 144 size_t result_stack_top;
pythontech 0:5868e8752d44 145 mp_parse_node_t *result_stack;
pythontech 0:5868e8752d44 146
pythontech 0:5868e8752d44 147 mp_lexer_t *lexer;
pythontech 0:5868e8752d44 148
pythontech 0:5868e8752d44 149 mp_parse_tree_t tree;
pythontech 0:5868e8752d44 150 mp_parse_chunk_t *cur_chunk;
pythontech 0:5868e8752d44 151
pythontech 0:5868e8752d44 152 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 153 mp_map_t consts;
pythontech 0:5868e8752d44 154 #endif
pythontech 0:5868e8752d44 155 } parser_t;
pythontech 0:5868e8752d44 156
pythontech 0:5868e8752d44 157 STATIC void *parser_alloc(parser_t *parser, size_t num_bytes) {
pythontech 0:5868e8752d44 158 // use a custom memory allocator to store parse nodes sequentially in large chunks
pythontech 0:5868e8752d44 159
pythontech 0:5868e8752d44 160 mp_parse_chunk_t *chunk = parser->cur_chunk;
pythontech 0:5868e8752d44 161
pythontech 0:5868e8752d44 162 if (chunk != NULL && chunk->union_.used + num_bytes > chunk->alloc) {
pythontech 0:5868e8752d44 163 // not enough room at end of previously allocated chunk so try to grow
pythontech 0:5868e8752d44 164 mp_parse_chunk_t *new_data = (mp_parse_chunk_t*)m_renew_maybe(byte, chunk,
pythontech 0:5868e8752d44 165 sizeof(mp_parse_chunk_t) + chunk->alloc,
pythontech 0:5868e8752d44 166 sizeof(mp_parse_chunk_t) + chunk->alloc + num_bytes, false);
pythontech 0:5868e8752d44 167 if (new_data == NULL) {
pythontech 0:5868e8752d44 168 // could not grow existing memory; shrink it to fit previous
pythontech 0:5868e8752d44 169 (void)m_renew_maybe(byte, chunk, sizeof(mp_parse_chunk_t) + chunk->alloc,
pythontech 0:5868e8752d44 170 sizeof(mp_parse_chunk_t) + chunk->union_.used, false);
pythontech 0:5868e8752d44 171 chunk->alloc = chunk->union_.used;
pythontech 0:5868e8752d44 172 chunk->union_.next = parser->tree.chunk;
pythontech 0:5868e8752d44 173 parser->tree.chunk = chunk;
pythontech 0:5868e8752d44 174 chunk = NULL;
pythontech 0:5868e8752d44 175 } else {
pythontech 0:5868e8752d44 176 // could grow existing memory
pythontech 0:5868e8752d44 177 chunk->alloc += num_bytes;
pythontech 0:5868e8752d44 178 }
pythontech 0:5868e8752d44 179 }
pythontech 0:5868e8752d44 180
pythontech 0:5868e8752d44 181 if (chunk == NULL) {
pythontech 0:5868e8752d44 182 // no previous chunk, allocate a new chunk
pythontech 0:5868e8752d44 183 size_t alloc = MICROPY_ALLOC_PARSE_CHUNK_INIT;
pythontech 0:5868e8752d44 184 if (alloc < num_bytes) {
pythontech 0:5868e8752d44 185 alloc = num_bytes;
pythontech 0:5868e8752d44 186 }
pythontech 0:5868e8752d44 187 chunk = (mp_parse_chunk_t*)m_new(byte, sizeof(mp_parse_chunk_t) + alloc);
pythontech 0:5868e8752d44 188 chunk->alloc = alloc;
pythontech 0:5868e8752d44 189 chunk->union_.used = 0;
pythontech 0:5868e8752d44 190 parser->cur_chunk = chunk;
pythontech 0:5868e8752d44 191 }
pythontech 0:5868e8752d44 192
pythontech 0:5868e8752d44 193 byte *ret = chunk->data + chunk->union_.used;
pythontech 0:5868e8752d44 194 chunk->union_.used += num_bytes;
pythontech 0:5868e8752d44 195 return ret;
pythontech 0:5868e8752d44 196 }
pythontech 0:5868e8752d44 197
pythontech 0:5868e8752d44 198 STATIC void push_rule(parser_t *parser, size_t src_line, const rule_t *rule, size_t arg_i) {
pythontech 0:5868e8752d44 199 if (parser->parse_error) {
pythontech 0:5868e8752d44 200 return;
pythontech 0:5868e8752d44 201 }
pythontech 0:5868e8752d44 202 if (parser->rule_stack_top >= parser->rule_stack_alloc) {
pythontech 0:5868e8752d44 203 rule_stack_t *rs = m_renew_maybe(rule_stack_t, parser->rule_stack, parser->rule_stack_alloc, parser->rule_stack_alloc + MICROPY_ALLOC_PARSE_RULE_INC, true);
pythontech 0:5868e8752d44 204 if (rs == NULL) {
pythontech 0:5868e8752d44 205 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 206 return;
pythontech 0:5868e8752d44 207 }
pythontech 0:5868e8752d44 208 parser->rule_stack = rs;
pythontech 0:5868e8752d44 209 parser->rule_stack_alloc += MICROPY_ALLOC_PARSE_RULE_INC;
pythontech 0:5868e8752d44 210 }
pythontech 0:5868e8752d44 211 rule_stack_t *rs = &parser->rule_stack[parser->rule_stack_top++];
pythontech 0:5868e8752d44 212 rs->src_line = src_line;
pythontech 0:5868e8752d44 213 rs->rule_id = rule->rule_id;
pythontech 0:5868e8752d44 214 rs->arg_i = arg_i;
pythontech 0:5868e8752d44 215 }
pythontech 0:5868e8752d44 216
pythontech 0:5868e8752d44 217 STATIC void push_rule_from_arg(parser_t *parser, size_t arg) {
pythontech 0:5868e8752d44 218 assert((arg & RULE_ARG_KIND_MASK) == RULE_ARG_RULE || (arg & RULE_ARG_KIND_MASK) == RULE_ARG_OPT_RULE);
pythontech 0:5868e8752d44 219 size_t rule_id = arg & RULE_ARG_ARG_MASK;
pythontech 0:5868e8752d44 220 assert(rule_id < RULE_maximum_number_of);
pythontech 0:5868e8752d44 221 push_rule(parser, parser->lexer->tok_line, rules[rule_id], 0);
pythontech 0:5868e8752d44 222 }
pythontech 0:5868e8752d44 223
pythontech 0:5868e8752d44 224 STATIC void pop_rule(parser_t *parser, const rule_t **rule, size_t *arg_i, size_t *src_line) {
pythontech 0:5868e8752d44 225 assert(!parser->parse_error);
pythontech 0:5868e8752d44 226 parser->rule_stack_top -= 1;
pythontech 0:5868e8752d44 227 *rule = rules[parser->rule_stack[parser->rule_stack_top].rule_id];
pythontech 0:5868e8752d44 228 *arg_i = parser->rule_stack[parser->rule_stack_top].arg_i;
pythontech 0:5868e8752d44 229 *src_line = parser->rule_stack[parser->rule_stack_top].src_line;
pythontech 0:5868e8752d44 230 }
pythontech 0:5868e8752d44 231
pythontech 0:5868e8752d44 232 mp_parse_node_t mp_parse_node_new_leaf(size_t kind, mp_int_t arg) {
pythontech 0:5868e8752d44 233 if (kind == MP_PARSE_NODE_SMALL_INT) {
pythontech 0:5868e8752d44 234 return (mp_parse_node_t)(kind | (arg << 1));
pythontech 0:5868e8752d44 235 }
pythontech 0:5868e8752d44 236 return (mp_parse_node_t)(kind | (arg << 4));
pythontech 0:5868e8752d44 237 }
pythontech 0:5868e8752d44 238
pythontech 0:5868e8752d44 239 bool mp_parse_node_get_int_maybe(mp_parse_node_t pn, mp_obj_t *o) {
pythontech 0:5868e8752d44 240 if (MP_PARSE_NODE_IS_SMALL_INT(pn)) {
pythontech 0:5868e8752d44 241 *o = MP_OBJ_NEW_SMALL_INT(MP_PARSE_NODE_LEAF_SMALL_INT(pn));
pythontech 0:5868e8752d44 242 return true;
pythontech 0:5868e8752d44 243 } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, RULE_const_object)) {
pythontech 0:5868e8752d44 244 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn;
pythontech 0:5868e8752d44 245 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 246 // nodes are 32-bit pointers, but need to extract 64-bit object
pythontech 0:5868e8752d44 247 *o = (uint64_t)pns->nodes[0] | ((uint64_t)pns->nodes[1] << 32);
pythontech 0:5868e8752d44 248 #else
pythontech 0:5868e8752d44 249 *o = (mp_obj_t)pns->nodes[0];
pythontech 0:5868e8752d44 250 #endif
pythontech 0:5868e8752d44 251 return MP_OBJ_IS_INT(*o);
pythontech 0:5868e8752d44 252 } else {
pythontech 0:5868e8752d44 253 return false;
pythontech 0:5868e8752d44 254 }
pythontech 0:5868e8752d44 255 }
pythontech 0:5868e8752d44 256
pythontech 0:5868e8752d44 257 int mp_parse_node_extract_list(mp_parse_node_t *pn, size_t pn_kind, mp_parse_node_t **nodes) {
pythontech 0:5868e8752d44 258 if (MP_PARSE_NODE_IS_NULL(*pn)) {
pythontech 0:5868e8752d44 259 *nodes = NULL;
pythontech 0:5868e8752d44 260 return 0;
pythontech 0:5868e8752d44 261 } else if (MP_PARSE_NODE_IS_LEAF(*pn)) {
pythontech 0:5868e8752d44 262 *nodes = pn;
pythontech 0:5868e8752d44 263 return 1;
pythontech 0:5868e8752d44 264 } else {
pythontech 0:5868e8752d44 265 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)(*pn);
pythontech 0:5868e8752d44 266 if (MP_PARSE_NODE_STRUCT_KIND(pns) != pn_kind) {
pythontech 0:5868e8752d44 267 *nodes = pn;
pythontech 0:5868e8752d44 268 return 1;
pythontech 0:5868e8752d44 269 } else {
pythontech 0:5868e8752d44 270 *nodes = pns->nodes;
pythontech 0:5868e8752d44 271 return MP_PARSE_NODE_STRUCT_NUM_NODES(pns);
pythontech 0:5868e8752d44 272 }
pythontech 0:5868e8752d44 273 }
pythontech 0:5868e8752d44 274 }
pythontech 0:5868e8752d44 275
pythontech 0:5868e8752d44 276 #if MICROPY_DEBUG_PRINTERS
pythontech 0:5868e8752d44 277 void mp_parse_node_print(mp_parse_node_t pn, size_t indent) {
pythontech 0:5868e8752d44 278 if (MP_PARSE_NODE_IS_STRUCT(pn)) {
pythontech 0:5868e8752d44 279 printf("[% 4d] ", (int)((mp_parse_node_struct_t*)pn)->source_line);
pythontech 0:5868e8752d44 280 } else {
pythontech 0:5868e8752d44 281 printf(" ");
pythontech 0:5868e8752d44 282 }
pythontech 0:5868e8752d44 283 for (size_t i = 0; i < indent; i++) {
pythontech 0:5868e8752d44 284 printf(" ");
pythontech 0:5868e8752d44 285 }
pythontech 0:5868e8752d44 286 if (MP_PARSE_NODE_IS_NULL(pn)) {
pythontech 0:5868e8752d44 287 printf("NULL\n");
pythontech 0:5868e8752d44 288 } else if (MP_PARSE_NODE_IS_SMALL_INT(pn)) {
pythontech 0:5868e8752d44 289 mp_int_t arg = MP_PARSE_NODE_LEAF_SMALL_INT(pn);
pythontech 0:5868e8752d44 290 printf("int(" INT_FMT ")\n", arg);
pythontech 0:5868e8752d44 291 } else if (MP_PARSE_NODE_IS_LEAF(pn)) {
pythontech 0:5868e8752d44 292 uintptr_t arg = MP_PARSE_NODE_LEAF_ARG(pn);
pythontech 0:5868e8752d44 293 switch (MP_PARSE_NODE_LEAF_KIND(pn)) {
pythontech 0:5868e8752d44 294 case MP_PARSE_NODE_ID: printf("id(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 295 case MP_PARSE_NODE_STRING: printf("str(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 296 case MP_PARSE_NODE_BYTES: printf("bytes(%s)\n", qstr_str(arg)); break;
pythontech 0:5868e8752d44 297 case MP_PARSE_NODE_TOKEN: printf("tok(%u)\n", (uint)arg); break;
pythontech 0:5868e8752d44 298 default: assert(0);
pythontech 0:5868e8752d44 299 }
pythontech 0:5868e8752d44 300 } else {
pythontech 0:5868e8752d44 301 // node must be a mp_parse_node_struct_t
pythontech 0:5868e8752d44 302 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn;
pythontech 0:5868e8752d44 303 if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_string) {
pythontech 0:5868e8752d44 304 printf("literal str(%.*s)\n", (int)pns->nodes[1], (char*)pns->nodes[0]);
pythontech 0:5868e8752d44 305 } else if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_bytes) {
pythontech 0:5868e8752d44 306 printf("literal bytes(%.*s)\n", (int)pns->nodes[1], (char*)pns->nodes[0]);
pythontech 0:5868e8752d44 307 } else if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_const_object) {
pythontech 0:5868e8752d44 308 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 309 printf("literal const(%016llx)\n", (uint64_t)pns->nodes[0] | ((uint64_t)pns->nodes[1] << 32));
pythontech 0:5868e8752d44 310 #else
pythontech 0:5868e8752d44 311 printf("literal const(%p)\n", (mp_obj_t)pns->nodes[0]);
pythontech 0:5868e8752d44 312 #endif
pythontech 0:5868e8752d44 313 } else {
pythontech 0:5868e8752d44 314 size_t n = MP_PARSE_NODE_STRUCT_NUM_NODES(pns);
pythontech 0:5868e8752d44 315 #ifdef USE_RULE_NAME
pythontech 0:5868e8752d44 316 printf("%s(%u) (n=%u)\n", rules[MP_PARSE_NODE_STRUCT_KIND(pns)]->rule_name, (uint)MP_PARSE_NODE_STRUCT_KIND(pns), (uint)n);
pythontech 0:5868e8752d44 317 #else
pythontech 0:5868e8752d44 318 printf("rule(%u) (n=%u)\n", (uint)MP_PARSE_NODE_STRUCT_KIND(pns), (uint)n);
pythontech 0:5868e8752d44 319 #endif
pythontech 0:5868e8752d44 320 for (size_t i = 0; i < n; i++) {
pythontech 0:5868e8752d44 321 mp_parse_node_print(pns->nodes[i], indent + 2);
pythontech 0:5868e8752d44 322 }
pythontech 0:5868e8752d44 323 }
pythontech 0:5868e8752d44 324 }
pythontech 0:5868e8752d44 325 }
pythontech 0:5868e8752d44 326 #endif // MICROPY_DEBUG_PRINTERS
pythontech 0:5868e8752d44 327
pythontech 0:5868e8752d44 328 /*
pythontech 0:5868e8752d44 329 STATIC void result_stack_show(parser_t *parser) {
pythontech 0:5868e8752d44 330 printf("result stack, most recent first\n");
pythontech 0:5868e8752d44 331 for (ssize_t i = parser->result_stack_top - 1; i >= 0; i--) {
pythontech 0:5868e8752d44 332 mp_parse_node_print(parser->result_stack[i], 0);
pythontech 0:5868e8752d44 333 }
pythontech 0:5868e8752d44 334 }
pythontech 0:5868e8752d44 335 */
pythontech 0:5868e8752d44 336
pythontech 0:5868e8752d44 337 STATIC mp_parse_node_t pop_result(parser_t *parser) {
pythontech 0:5868e8752d44 338 if (parser->parse_error) {
pythontech 0:5868e8752d44 339 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 340 }
pythontech 0:5868e8752d44 341 assert(parser->result_stack_top > 0);
pythontech 0:5868e8752d44 342 return parser->result_stack[--parser->result_stack_top];
pythontech 0:5868e8752d44 343 }
pythontech 0:5868e8752d44 344
pythontech 0:5868e8752d44 345 STATIC mp_parse_node_t peek_result(parser_t *parser, size_t pos) {
pythontech 0:5868e8752d44 346 if (parser->parse_error) {
pythontech 0:5868e8752d44 347 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 348 }
pythontech 0:5868e8752d44 349 assert(parser->result_stack_top > pos);
pythontech 0:5868e8752d44 350 return parser->result_stack[parser->result_stack_top - 1 - pos];
pythontech 0:5868e8752d44 351 }
pythontech 0:5868e8752d44 352
pythontech 0:5868e8752d44 353 STATIC void push_result_node(parser_t *parser, mp_parse_node_t pn) {
pythontech 0:5868e8752d44 354 if (parser->parse_error) {
pythontech 0:5868e8752d44 355 return;
pythontech 0:5868e8752d44 356 }
pythontech 0:5868e8752d44 357 if (parser->result_stack_top >= parser->result_stack_alloc) {
pythontech 0:5868e8752d44 358 mp_parse_node_t *stack = m_renew_maybe(mp_parse_node_t, parser->result_stack, parser->result_stack_alloc, parser->result_stack_alloc + MICROPY_ALLOC_PARSE_RESULT_INC, true);
pythontech 0:5868e8752d44 359 if (stack == NULL) {
pythontech 0:5868e8752d44 360 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 361 return;
pythontech 0:5868e8752d44 362 }
pythontech 0:5868e8752d44 363 parser->result_stack = stack;
pythontech 0:5868e8752d44 364 parser->result_stack_alloc += MICROPY_ALLOC_PARSE_RESULT_INC;
pythontech 0:5868e8752d44 365 }
pythontech 0:5868e8752d44 366 parser->result_stack[parser->result_stack_top++] = pn;
pythontech 0:5868e8752d44 367 }
pythontech 0:5868e8752d44 368
pythontech 0:5868e8752d44 369 STATIC mp_parse_node_t make_node_string_bytes(parser_t *parser, size_t src_line, size_t rule_kind, const char *str, size_t len) {
pythontech 0:5868e8752d44 370 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_parse_node_t) * 2);
pythontech 0:5868e8752d44 371 if (pn == NULL) {
pythontech 0:5868e8752d44 372 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 373 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 374 }
pythontech 0:5868e8752d44 375 pn->source_line = src_line;
pythontech 0:5868e8752d44 376 pn->kind_num_nodes = rule_kind | (2 << 8);
pythontech 0:5868e8752d44 377 char *p = m_new(char, len);
pythontech 0:5868e8752d44 378 memcpy(p, str, len);
pythontech 0:5868e8752d44 379 pn->nodes[0] = (uintptr_t)p;
pythontech 0:5868e8752d44 380 pn->nodes[1] = len;
pythontech 0:5868e8752d44 381 return (mp_parse_node_t)pn;
pythontech 0:5868e8752d44 382 }
pythontech 0:5868e8752d44 383
pythontech 0:5868e8752d44 384 STATIC mp_parse_node_t make_node_const_object(parser_t *parser, size_t src_line, mp_obj_t obj) {
pythontech 0:5868e8752d44 385 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_obj_t));
pythontech 0:5868e8752d44 386 if (pn == NULL) {
pythontech 0:5868e8752d44 387 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 388 return MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 389 }
pythontech 0:5868e8752d44 390 pn->source_line = src_line;
pythontech 0:5868e8752d44 391 #if MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D
pythontech 0:5868e8752d44 392 // nodes are 32-bit pointers, but need to store 64-bit object
pythontech 0:5868e8752d44 393 pn->kind_num_nodes = RULE_const_object | (2 << 8);
pythontech 0:5868e8752d44 394 pn->nodes[0] = (uint64_t)obj;
pythontech 0:5868e8752d44 395 pn->nodes[1] = (uint64_t)obj >> 32;
pythontech 0:5868e8752d44 396 #else
pythontech 0:5868e8752d44 397 pn->kind_num_nodes = RULE_const_object | (1 << 8);
pythontech 0:5868e8752d44 398 pn->nodes[0] = (uintptr_t)obj;
pythontech 0:5868e8752d44 399 #endif
pythontech 0:5868e8752d44 400 return (mp_parse_node_t)pn;
pythontech 0:5868e8752d44 401 }
pythontech 0:5868e8752d44 402
pythontech 0:5868e8752d44 403 STATIC void push_result_token(parser_t *parser) {
pythontech 0:5868e8752d44 404 mp_parse_node_t pn;
pythontech 0:5868e8752d44 405 mp_lexer_t *lex = parser->lexer;
pythontech 0:5868e8752d44 406 if (lex->tok_kind == MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 407 qstr id = qstr_from_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 408 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 409 // lookup identifier in table of dynamic constants
pythontech 0:5868e8752d44 410 mp_map_elem_t *elem = mp_map_lookup(&parser->consts, MP_OBJ_NEW_QSTR(id), MP_MAP_LOOKUP);
pythontech 0:5868e8752d44 411 if (elem != NULL) {
pythontech 0:5868e8752d44 412 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(elem->value));
pythontech 0:5868e8752d44 413 } else
pythontech 0:5868e8752d44 414 #endif
pythontech 0:5868e8752d44 415 {
pythontech 0:5868e8752d44 416 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_ID, id);
pythontech 0:5868e8752d44 417 }
pythontech 0:5868e8752d44 418 } else if (lex->tok_kind == MP_TOKEN_INTEGER) {
pythontech 0:5868e8752d44 419 mp_obj_t o = mp_parse_num_integer(lex->vstr.buf, lex->vstr.len, 0, lex);
pythontech 0:5868e8752d44 420 if (MP_OBJ_IS_SMALL_INT(o)) {
pythontech 0:5868e8752d44 421 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(o));
pythontech 0:5868e8752d44 422 } else {
pythontech 0:5868e8752d44 423 pn = make_node_const_object(parser, lex->tok_line, o);
pythontech 0:5868e8752d44 424 }
pythontech 0:5868e8752d44 425 } else if (lex->tok_kind == MP_TOKEN_FLOAT_OR_IMAG) {
pythontech 0:5868e8752d44 426 mp_obj_t o = mp_parse_num_decimal(lex->vstr.buf, lex->vstr.len, true, false, lex);
pythontech 0:5868e8752d44 427 pn = make_node_const_object(parser, lex->tok_line, o);
pythontech 0:5868e8752d44 428 } else if (lex->tok_kind == MP_TOKEN_STRING || lex->tok_kind == MP_TOKEN_BYTES) {
pythontech 0:5868e8752d44 429 // Don't automatically intern all strings/bytes. doc strings (which are usually large)
pythontech 0:5868e8752d44 430 // will be discarded by the compiler, and so we shouldn't intern them.
pythontech 0:5868e8752d44 431 qstr qst = MP_QSTR_NULL;
pythontech 0:5868e8752d44 432 if (lex->vstr.len <= MICROPY_ALLOC_PARSE_INTERN_STRING_LEN) {
pythontech 0:5868e8752d44 433 // intern short strings
pythontech 0:5868e8752d44 434 qst = qstr_from_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 435 } else {
pythontech 0:5868e8752d44 436 // check if this string is already interned
pythontech 0:5868e8752d44 437 qst = qstr_find_strn(lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 438 }
pythontech 0:5868e8752d44 439 if (qst != MP_QSTR_NULL) {
pythontech 0:5868e8752d44 440 // qstr exists, make a leaf node
pythontech 0:5868e8752d44 441 pn = mp_parse_node_new_leaf(lex->tok_kind == MP_TOKEN_STRING ? MP_PARSE_NODE_STRING : MP_PARSE_NODE_BYTES, qst);
pythontech 0:5868e8752d44 442 } else {
pythontech 0:5868e8752d44 443 // not interned, make a node holding a pointer to the string/bytes data
pythontech 0:5868e8752d44 444 pn = make_node_string_bytes(parser, lex->tok_line, lex->tok_kind == MP_TOKEN_STRING ? RULE_string : RULE_bytes, lex->vstr.buf, lex->vstr.len);
pythontech 0:5868e8752d44 445 }
pythontech 0:5868e8752d44 446 } else {
pythontech 0:5868e8752d44 447 pn = mp_parse_node_new_leaf(MP_PARSE_NODE_TOKEN, lex->tok_kind);
pythontech 0:5868e8752d44 448 }
pythontech 0:5868e8752d44 449 push_result_node(parser, pn);
pythontech 0:5868e8752d44 450 }
pythontech 0:5868e8752d44 451
pythontech 0:5868e8752d44 452 #if MICROPY_COMP_MODULE_CONST
pythontech 0:5868e8752d44 453 STATIC const mp_rom_map_elem_t mp_constants_table[] = {
pythontech 0:5868e8752d44 454 #if MICROPY_PY_UCTYPES
pythontech 0:5868e8752d44 455 { MP_ROM_QSTR(MP_QSTR_uctypes), MP_ROM_PTR(&mp_module_uctypes) },
pythontech 0:5868e8752d44 456 #endif
pythontech 0:5868e8752d44 457 // Extra constants as defined by a port
pythontech 0:5868e8752d44 458 MICROPY_PORT_CONSTANTS
pythontech 0:5868e8752d44 459 };
pythontech 0:5868e8752d44 460 STATIC MP_DEFINE_CONST_MAP(mp_constants_map, mp_constants_table);
pythontech 0:5868e8752d44 461 #endif
pythontech 0:5868e8752d44 462
pythontech 0:5868e8752d44 463 #if MICROPY_COMP_CONST_FOLDING
pythontech 0:5868e8752d44 464 STATIC bool fold_constants(parser_t *parser, const rule_t *rule, size_t num_args) {
pythontech 0:5868e8752d44 465 // this code does folding of arbitrary integer expressions, eg 1 + 2 * 3 + 4
pythontech 0:5868e8752d44 466 // it does not do partial folding, eg 1 + 2 + x -> 3 + x
pythontech 0:5868e8752d44 467
pythontech 0:5868e8752d44 468 mp_obj_t arg0;
pythontech 0:5868e8752d44 469 if (rule->rule_id == RULE_expr
pythontech 0:5868e8752d44 470 || rule->rule_id == RULE_xor_expr
pythontech 0:5868e8752d44 471 || rule->rule_id == RULE_and_expr) {
pythontech 0:5868e8752d44 472 // folding for binary ops: | ^ &
pythontech 0:5868e8752d44 473 mp_parse_node_t pn = peek_result(parser, num_args - 1);
pythontech 0:5868e8752d44 474 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 475 return false;
pythontech 0:5868e8752d44 476 }
pythontech 0:5868e8752d44 477 mp_binary_op_t op;
pythontech 0:5868e8752d44 478 if (rule->rule_id == RULE_expr) {
pythontech 0:5868e8752d44 479 op = MP_BINARY_OP_OR;
pythontech 0:5868e8752d44 480 } else if (rule->rule_id == RULE_xor_expr) {
pythontech 0:5868e8752d44 481 op = MP_BINARY_OP_XOR;
pythontech 0:5868e8752d44 482 } else {
pythontech 0:5868e8752d44 483 op = MP_BINARY_OP_AND;
pythontech 0:5868e8752d44 484 }
pythontech 0:5868e8752d44 485 for (ssize_t i = num_args - 2; i >= 0; --i) {
pythontech 0:5868e8752d44 486 pn = peek_result(parser, i);
pythontech 0:5868e8752d44 487 mp_obj_t arg1;
pythontech 0:5868e8752d44 488 if (!mp_parse_node_get_int_maybe(pn, &arg1)) {
pythontech 0:5868e8752d44 489 return false;
pythontech 0:5868e8752d44 490 }
pythontech 0:5868e8752d44 491 arg0 = mp_binary_op(op, arg0, arg1);
pythontech 0:5868e8752d44 492 }
pythontech 0:5868e8752d44 493 } else if (rule->rule_id == RULE_shift_expr
pythontech 0:5868e8752d44 494 || rule->rule_id == RULE_arith_expr
pythontech 0:5868e8752d44 495 || rule->rule_id == RULE_term) {
pythontech 0:5868e8752d44 496 // folding for binary ops: << >> + - * / % //
pythontech 0:5868e8752d44 497 mp_parse_node_t pn = peek_result(parser, num_args - 1);
pythontech 0:5868e8752d44 498 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 499 return false;
pythontech 0:5868e8752d44 500 }
pythontech 0:5868e8752d44 501 for (ssize_t i = num_args - 2; i >= 1; i -= 2) {
pythontech 0:5868e8752d44 502 pn = peek_result(parser, i - 1);
pythontech 0:5868e8752d44 503 mp_obj_t arg1;
pythontech 0:5868e8752d44 504 if (!mp_parse_node_get_int_maybe(pn, &arg1)) {
pythontech 0:5868e8752d44 505 return false;
pythontech 0:5868e8752d44 506 }
pythontech 0:5868e8752d44 507 mp_token_kind_t tok = MP_PARSE_NODE_LEAF_ARG(peek_result(parser, i));
pythontech 0:5868e8752d44 508 static const uint8_t token_to_op[] = {
pythontech 0:5868e8752d44 509 MP_BINARY_OP_ADD,
pythontech 0:5868e8752d44 510 MP_BINARY_OP_SUBTRACT,
pythontech 0:5868e8752d44 511 MP_BINARY_OP_MULTIPLY,
pythontech 0:5868e8752d44 512 255,//MP_BINARY_OP_POWER,
pythontech 0:5868e8752d44 513 255,//MP_BINARY_OP_TRUE_DIVIDE,
pythontech 0:5868e8752d44 514 MP_BINARY_OP_FLOOR_DIVIDE,
pythontech 0:5868e8752d44 515 MP_BINARY_OP_MODULO,
pythontech 0:5868e8752d44 516 255,//MP_BINARY_OP_LESS
pythontech 0:5868e8752d44 517 MP_BINARY_OP_LSHIFT,
pythontech 0:5868e8752d44 518 255,//MP_BINARY_OP_MORE
pythontech 0:5868e8752d44 519 MP_BINARY_OP_RSHIFT,
pythontech 0:5868e8752d44 520 };
pythontech 0:5868e8752d44 521 mp_binary_op_t op = token_to_op[tok - MP_TOKEN_OP_PLUS];
pythontech 0:5868e8752d44 522 if (op == (mp_binary_op_t)255) {
pythontech 0:5868e8752d44 523 return false;
pythontech 0:5868e8752d44 524 }
pythontech 0:5868e8752d44 525 int rhs_sign = mp_obj_int_sign(arg1);
pythontech 0:5868e8752d44 526 if (op <= MP_BINARY_OP_RSHIFT) {
pythontech 0:5868e8752d44 527 // << and >> can't have negative rhs
pythontech 0:5868e8752d44 528 if (rhs_sign < 0) {
pythontech 0:5868e8752d44 529 return false;
pythontech 0:5868e8752d44 530 }
pythontech 0:5868e8752d44 531 } else if (op >= MP_BINARY_OP_FLOOR_DIVIDE) {
pythontech 0:5868e8752d44 532 // % and // can't have zero rhs
pythontech 0:5868e8752d44 533 if (rhs_sign == 0) {
pythontech 0:5868e8752d44 534 return false;
pythontech 0:5868e8752d44 535 }
pythontech 0:5868e8752d44 536 }
pythontech 0:5868e8752d44 537 arg0 = mp_binary_op(op, arg0, arg1);
pythontech 0:5868e8752d44 538 }
pythontech 0:5868e8752d44 539 } else if (rule->rule_id == RULE_factor_2) {
pythontech 0:5868e8752d44 540 // folding for unary ops: + - ~
pythontech 0:5868e8752d44 541 mp_parse_node_t pn = peek_result(parser, 0);
pythontech 0:5868e8752d44 542 if (!mp_parse_node_get_int_maybe(pn, &arg0)) {
pythontech 0:5868e8752d44 543 return false;
pythontech 0:5868e8752d44 544 }
pythontech 0:5868e8752d44 545 mp_token_kind_t tok = MP_PARSE_NODE_LEAF_ARG(peek_result(parser, 1));
pythontech 0:5868e8752d44 546 mp_unary_op_t op;
pythontech 0:5868e8752d44 547 if (tok == MP_TOKEN_OP_PLUS) {
pythontech 0:5868e8752d44 548 op = MP_UNARY_OP_POSITIVE;
pythontech 0:5868e8752d44 549 } else if (tok == MP_TOKEN_OP_MINUS) {
pythontech 0:5868e8752d44 550 op = MP_UNARY_OP_NEGATIVE;
pythontech 0:5868e8752d44 551 } else {
pythontech 0:5868e8752d44 552 assert(tok == MP_TOKEN_OP_TILDE); // should be
pythontech 0:5868e8752d44 553 op = MP_UNARY_OP_INVERT;
pythontech 0:5868e8752d44 554 }
pythontech 0:5868e8752d44 555 arg0 = mp_unary_op(op, arg0);
pythontech 0:5868e8752d44 556
pythontech 0:5868e8752d44 557 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 558 } else if (rule->rule_id == RULE_expr_stmt) {
pythontech 0:5868e8752d44 559 mp_parse_node_t pn1 = peek_result(parser, 0);
pythontech 0:5868e8752d44 560 if (!MP_PARSE_NODE_IS_NULL(pn1)
pythontech 0:5868e8752d44 561 && !(MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_expr_stmt_augassign)
pythontech 0:5868e8752d44 562 || MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_expr_stmt_assign_list))) {
pythontech 0:5868e8752d44 563 // this node is of the form <x> = <y>
pythontech 0:5868e8752d44 564 mp_parse_node_t pn0 = peek_result(parser, 1);
pythontech 0:5868e8752d44 565 if (MP_PARSE_NODE_IS_ID(pn0)
pythontech 0:5868e8752d44 566 && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_power)
pythontech 0:5868e8752d44 567 && MP_PARSE_NODE_IS_ID(((mp_parse_node_struct_t*)pn1)->nodes[0])
pythontech 0:5868e8752d44 568 && MP_PARSE_NODE_LEAF_ARG(((mp_parse_node_struct_t*)pn1)->nodes[0]) == MP_QSTR_const
pythontech 0:5868e8752d44 569 && MP_PARSE_NODE_IS_STRUCT_KIND(((mp_parse_node_struct_t*)pn1)->nodes[1], RULE_trailer_paren)
pythontech 0:5868e8752d44 570 && MP_PARSE_NODE_IS_NULL(((mp_parse_node_struct_t*)pn1)->nodes[2])
pythontech 0:5868e8752d44 571 ) {
pythontech 0:5868e8752d44 572 // code to assign dynamic constants: id = const(value)
pythontech 0:5868e8752d44 573
pythontech 0:5868e8752d44 574 // get the id
pythontech 0:5868e8752d44 575 qstr id = MP_PARSE_NODE_LEAF_ARG(pn0);
pythontech 0:5868e8752d44 576
pythontech 0:5868e8752d44 577 // get the value
pythontech 0:5868e8752d44 578 mp_parse_node_t pn_value = ((mp_parse_node_struct_t*)((mp_parse_node_struct_t*)pn1)->nodes[1])->nodes[0];
pythontech 0:5868e8752d44 579 if (!MP_PARSE_NODE_IS_SMALL_INT(pn_value)) {
pythontech 0:5868e8752d44 580 parser->parse_error = PARSE_ERROR_CONST;
pythontech 0:5868e8752d44 581 return false;
pythontech 0:5868e8752d44 582 }
pythontech 0:5868e8752d44 583 mp_int_t value = MP_PARSE_NODE_LEAF_SMALL_INT(pn_value);
pythontech 0:5868e8752d44 584
pythontech 0:5868e8752d44 585 // store the value in the table of dynamic constants
pythontech 0:5868e8752d44 586 mp_map_elem_t *elem = mp_map_lookup(&parser->consts, MP_OBJ_NEW_QSTR(id), MP_MAP_LOOKUP_ADD_IF_NOT_FOUND);
pythontech 0:5868e8752d44 587 assert(elem->value == MP_OBJ_NULL);
pythontech 0:5868e8752d44 588 elem->value = MP_OBJ_NEW_SMALL_INT(value);
pythontech 0:5868e8752d44 589
pythontech 0:5868e8752d44 590 // replace const(value) with value
pythontech 0:5868e8752d44 591 pop_result(parser);
pythontech 0:5868e8752d44 592 push_result_node(parser, pn_value);
pythontech 0:5868e8752d44 593
pythontech 0:5868e8752d44 594 // finished folding this assignment, but we still want it to be part of the tree
pythontech 0:5868e8752d44 595 return false;
pythontech 0:5868e8752d44 596 }
pythontech 0:5868e8752d44 597 }
pythontech 0:5868e8752d44 598 return false;
pythontech 0:5868e8752d44 599 #endif
pythontech 0:5868e8752d44 600
pythontech 0:5868e8752d44 601 #if MICROPY_COMP_MODULE_CONST
pythontech 0:5868e8752d44 602 } else if (rule->rule_id == RULE_power) {
pythontech 0:5868e8752d44 603 mp_parse_node_t pn0 = peek_result(parser, 2);
pythontech 0:5868e8752d44 604 mp_parse_node_t pn1 = peek_result(parser, 1);
pythontech 0:5868e8752d44 605 mp_parse_node_t pn2 = peek_result(parser, 0);
pythontech 0:5868e8752d44 606 if (!(MP_PARSE_NODE_IS_ID(pn0)
pythontech 0:5868e8752d44 607 && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_trailer_period)
pythontech 0:5868e8752d44 608 && MP_PARSE_NODE_IS_NULL(pn2))) {
pythontech 0:5868e8752d44 609 return false;
pythontech 0:5868e8752d44 610 }
pythontech 0:5868e8752d44 611 // id1.id2
pythontech 0:5868e8752d44 612 // look it up in constant table, see if it can be replaced with an integer
pythontech 0:5868e8752d44 613 mp_parse_node_struct_t *pns1 = (mp_parse_node_struct_t*)pn1;
pythontech 0:5868e8752d44 614 assert(MP_PARSE_NODE_IS_ID(pns1->nodes[0]));
pythontech 0:5868e8752d44 615 qstr q_base = MP_PARSE_NODE_LEAF_ARG(pn0);
pythontech 0:5868e8752d44 616 qstr q_attr = MP_PARSE_NODE_LEAF_ARG(pns1->nodes[0]);
pythontech 0:5868e8752d44 617 mp_map_elem_t *elem = mp_map_lookup((mp_map_t*)&mp_constants_map, MP_OBJ_NEW_QSTR(q_base), MP_MAP_LOOKUP);
pythontech 0:5868e8752d44 618 if (elem == NULL) {
pythontech 0:5868e8752d44 619 return false;
pythontech 0:5868e8752d44 620 }
pythontech 0:5868e8752d44 621 mp_obj_t dest[2];
pythontech 0:5868e8752d44 622 mp_load_method_maybe(elem->value, q_attr, dest);
pythontech 0:5868e8752d44 623 if (!(dest[0] != MP_OBJ_NULL && MP_OBJ_IS_INT(dest[0]) && dest[1] == MP_OBJ_NULL)) {
pythontech 0:5868e8752d44 624 return false;
pythontech 0:5868e8752d44 625 }
pythontech 0:5868e8752d44 626 arg0 = dest[0];
pythontech 0:5868e8752d44 627 #endif
pythontech 0:5868e8752d44 628
pythontech 0:5868e8752d44 629 } else {
pythontech 0:5868e8752d44 630 return false;
pythontech 0:5868e8752d44 631 }
pythontech 0:5868e8752d44 632
pythontech 0:5868e8752d44 633 // success folding this rule
pythontech 0:5868e8752d44 634
pythontech 0:5868e8752d44 635 for (size_t i = num_args; i > 0; i--) {
pythontech 0:5868e8752d44 636 pop_result(parser);
pythontech 0:5868e8752d44 637 }
pythontech 0:5868e8752d44 638 if (MP_OBJ_IS_SMALL_INT(arg0)) {
pythontech 0:5868e8752d44 639 push_result_node(parser, mp_parse_node_new_leaf(MP_PARSE_NODE_SMALL_INT, MP_OBJ_SMALL_INT_VALUE(arg0)));
pythontech 0:5868e8752d44 640 } else {
pythontech 0:5868e8752d44 641 // TODO reuse memory for parse node struct?
pythontech 0:5868e8752d44 642 push_result_node(parser, make_node_const_object(parser, 0, arg0));
pythontech 0:5868e8752d44 643 }
pythontech 0:5868e8752d44 644
pythontech 0:5868e8752d44 645 return true;
pythontech 0:5868e8752d44 646 }
pythontech 0:5868e8752d44 647 #endif
pythontech 0:5868e8752d44 648
pythontech 0:5868e8752d44 649 STATIC void push_result_rule(parser_t *parser, size_t src_line, const rule_t *rule, size_t num_args) {
pythontech 0:5868e8752d44 650 // optimise away parenthesis around an expression if possible
pythontech 0:5868e8752d44 651 if (rule->rule_id == RULE_atom_paren) {
pythontech 0:5868e8752d44 652 // there should be just 1 arg for this rule
pythontech 0:5868e8752d44 653 mp_parse_node_t pn = peek_result(parser, 0);
pythontech 0:5868e8752d44 654 if (MP_PARSE_NODE_IS_NULL(pn)) {
pythontech 0:5868e8752d44 655 // need to keep parenthesis for ()
pythontech 0:5868e8752d44 656 } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, RULE_testlist_comp)) {
pythontech 0:5868e8752d44 657 // need to keep parenthesis for (a, b, ...)
pythontech 0:5868e8752d44 658 } else {
pythontech 0:5868e8752d44 659 // parenthesis around a single expression, so it's just the expression
pythontech 0:5868e8752d44 660 return;
pythontech 0:5868e8752d44 661 }
pythontech 0:5868e8752d44 662 }
pythontech 0:5868e8752d44 663
pythontech 0:5868e8752d44 664 #if MICROPY_COMP_CONST_FOLDING
pythontech 0:5868e8752d44 665 if (fold_constants(parser, rule, num_args)) {
pythontech 0:5868e8752d44 666 // we folded this rule so return straight away
pythontech 0:5868e8752d44 667 return;
pythontech 0:5868e8752d44 668 }
pythontech 0:5868e8752d44 669 #endif
pythontech 0:5868e8752d44 670
pythontech 0:5868e8752d44 671 mp_parse_node_struct_t *pn = parser_alloc(parser, sizeof(mp_parse_node_struct_t) + sizeof(mp_parse_node_t) * num_args);
pythontech 0:5868e8752d44 672 if (pn == NULL) {
pythontech 0:5868e8752d44 673 parser->parse_error = PARSE_ERROR_MEMORY;
pythontech 0:5868e8752d44 674 return;
pythontech 0:5868e8752d44 675 }
pythontech 0:5868e8752d44 676 pn->source_line = src_line;
pythontech 0:5868e8752d44 677 pn->kind_num_nodes = (rule->rule_id & 0xff) | (num_args << 8);
pythontech 0:5868e8752d44 678 for (size_t i = num_args; i > 0; i--) {
pythontech 0:5868e8752d44 679 pn->nodes[i - 1] = pop_result(parser);
pythontech 0:5868e8752d44 680 }
pythontech 0:5868e8752d44 681 push_result_node(parser, (mp_parse_node_t)pn);
pythontech 0:5868e8752d44 682 }
pythontech 0:5868e8752d44 683
pythontech 0:5868e8752d44 684 mp_parse_tree_t mp_parse(mp_lexer_t *lex, mp_parse_input_kind_t input_kind) {
pythontech 0:5868e8752d44 685
pythontech 0:5868e8752d44 686 // initialise parser and allocate memory for its stacks
pythontech 0:5868e8752d44 687
pythontech 0:5868e8752d44 688 parser_t parser;
pythontech 0:5868e8752d44 689
pythontech 0:5868e8752d44 690 parser.parse_error = PARSE_ERROR_NONE;
pythontech 0:5868e8752d44 691
pythontech 0:5868e8752d44 692 parser.rule_stack_alloc = MICROPY_ALLOC_PARSE_RULE_INIT;
pythontech 0:5868e8752d44 693 parser.rule_stack_top = 0;
pythontech 0:5868e8752d44 694 parser.rule_stack = m_new_maybe(rule_stack_t, parser.rule_stack_alloc);
pythontech 0:5868e8752d44 695
pythontech 0:5868e8752d44 696 parser.result_stack_alloc = MICROPY_ALLOC_PARSE_RESULT_INIT;
pythontech 0:5868e8752d44 697 parser.result_stack_top = 0;
pythontech 0:5868e8752d44 698 parser.result_stack = m_new_maybe(mp_parse_node_t, parser.result_stack_alloc);
pythontech 0:5868e8752d44 699
pythontech 0:5868e8752d44 700 parser.lexer = lex;
pythontech 0:5868e8752d44 701
pythontech 0:5868e8752d44 702 parser.tree.chunk = NULL;
pythontech 0:5868e8752d44 703 parser.cur_chunk = NULL;
pythontech 0:5868e8752d44 704
pythontech 0:5868e8752d44 705 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 706 mp_map_init(&parser.consts, 0);
pythontech 0:5868e8752d44 707 #endif
pythontech 0:5868e8752d44 708
pythontech 0:5868e8752d44 709 // check if we could allocate the stacks
pythontech 0:5868e8752d44 710 if (parser.rule_stack == NULL || parser.result_stack == NULL) {
pythontech 0:5868e8752d44 711 goto memory_error;
pythontech 0:5868e8752d44 712 }
pythontech 0:5868e8752d44 713
pythontech 0:5868e8752d44 714 // work out the top-level rule to use, and push it on the stack
pythontech 0:5868e8752d44 715 size_t top_level_rule;
pythontech 0:5868e8752d44 716 switch (input_kind) {
pythontech 0:5868e8752d44 717 case MP_PARSE_SINGLE_INPUT: top_level_rule = RULE_single_input; break;
pythontech 0:5868e8752d44 718 case MP_PARSE_EVAL_INPUT: top_level_rule = RULE_eval_input; break;
pythontech 0:5868e8752d44 719 default: top_level_rule = RULE_file_input;
pythontech 0:5868e8752d44 720 }
pythontech 0:5868e8752d44 721 push_rule(&parser, lex->tok_line, rules[top_level_rule], 0);
pythontech 0:5868e8752d44 722
pythontech 0:5868e8752d44 723 // parse!
pythontech 0:5868e8752d44 724
pythontech 0:5868e8752d44 725 size_t n, i; // state for the current rule
pythontech 0:5868e8752d44 726 size_t rule_src_line; // source line for the first token matched by the current rule
pythontech 0:5868e8752d44 727 bool backtrack = false;
pythontech 0:5868e8752d44 728 const rule_t *rule = NULL;
pythontech 0:5868e8752d44 729
pythontech 0:5868e8752d44 730 for (;;) {
pythontech 0:5868e8752d44 731 next_rule:
pythontech 0:5868e8752d44 732 if (parser.rule_stack_top == 0 || parser.parse_error) {
pythontech 0:5868e8752d44 733 break;
pythontech 0:5868e8752d44 734 }
pythontech 0:5868e8752d44 735
pythontech 0:5868e8752d44 736 pop_rule(&parser, &rule, &i, &rule_src_line);
pythontech 0:5868e8752d44 737 n = rule->act & RULE_ACT_ARG_MASK;
pythontech 0:5868e8752d44 738
pythontech 0:5868e8752d44 739 /*
pythontech 0:5868e8752d44 740 // debugging
pythontech 0:5868e8752d44 741 printf("depth=%d ", parser.rule_stack_top);
pythontech 0:5868e8752d44 742 for (int j = 0; j < parser.rule_stack_top; ++j) {
pythontech 0:5868e8752d44 743 printf(" ");
pythontech 0:5868e8752d44 744 }
pythontech 0:5868e8752d44 745 printf("%s n=%d i=%d bt=%d\n", rule->rule_name, n, i, backtrack);
pythontech 0:5868e8752d44 746 */
pythontech 0:5868e8752d44 747
pythontech 0:5868e8752d44 748 switch (rule->act & RULE_ACT_KIND_MASK) {
pythontech 0:5868e8752d44 749 case RULE_ACT_OR:
pythontech 0:5868e8752d44 750 if (i > 0 && !backtrack) {
pythontech 0:5868e8752d44 751 goto next_rule;
pythontech 0:5868e8752d44 752 } else {
pythontech 0:5868e8752d44 753 backtrack = false;
pythontech 0:5868e8752d44 754 }
pythontech 0:5868e8752d44 755 for (; i < n; ++i) {
pythontech 0:5868e8752d44 756 uint16_t kind = rule->arg[i] & RULE_ARG_KIND_MASK;
pythontech 0:5868e8752d44 757 if (kind == RULE_ARG_TOK) {
pythontech 0:5868e8752d44 758 if (lex->tok_kind == (rule->arg[i] & RULE_ARG_ARG_MASK)) {
pythontech 0:5868e8752d44 759 push_result_token(&parser);
pythontech 0:5868e8752d44 760 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 761 goto next_rule;
pythontech 0:5868e8752d44 762 }
pythontech 0:5868e8752d44 763 } else {
pythontech 0:5868e8752d44 764 assert(kind == RULE_ARG_RULE);
pythontech 0:5868e8752d44 765 if (i + 1 < n) {
pythontech 0:5868e8752d44 766 push_rule(&parser, rule_src_line, rule, i + 1); // save this or-rule
pythontech 0:5868e8752d44 767 }
pythontech 0:5868e8752d44 768 push_rule_from_arg(&parser, rule->arg[i]); // push child of or-rule
pythontech 0:5868e8752d44 769 goto next_rule;
pythontech 0:5868e8752d44 770 }
pythontech 0:5868e8752d44 771 }
pythontech 0:5868e8752d44 772 backtrack = true;
pythontech 0:5868e8752d44 773 break;
pythontech 0:5868e8752d44 774
pythontech 0:5868e8752d44 775 case RULE_ACT_AND: {
pythontech 0:5868e8752d44 776
pythontech 0:5868e8752d44 777 // failed, backtrack if we can, else syntax error
pythontech 0:5868e8752d44 778 if (backtrack) {
pythontech 0:5868e8752d44 779 assert(i > 0);
pythontech 0:5868e8752d44 780 if ((rule->arg[i - 1] & RULE_ARG_KIND_MASK) == RULE_ARG_OPT_RULE) {
pythontech 0:5868e8752d44 781 // an optional rule that failed, so continue with next arg
pythontech 0:5868e8752d44 782 push_result_node(&parser, MP_PARSE_NODE_NULL);
pythontech 0:5868e8752d44 783 backtrack = false;
pythontech 0:5868e8752d44 784 } else {
pythontech 0:5868e8752d44 785 // a mandatory rule that failed, so propagate backtrack
pythontech 0:5868e8752d44 786 if (i > 1) {
pythontech 0:5868e8752d44 787 // already eaten tokens so can't backtrack
pythontech 0:5868e8752d44 788 goto syntax_error;
pythontech 0:5868e8752d44 789 } else {
pythontech 0:5868e8752d44 790 goto next_rule;
pythontech 0:5868e8752d44 791 }
pythontech 0:5868e8752d44 792 }
pythontech 0:5868e8752d44 793 }
pythontech 0:5868e8752d44 794
pythontech 0:5868e8752d44 795 // progress through the rule
pythontech 0:5868e8752d44 796 for (; i < n; ++i) {
pythontech 0:5868e8752d44 797 switch (rule->arg[i] & RULE_ARG_KIND_MASK) {
pythontech 0:5868e8752d44 798 case RULE_ARG_TOK: {
pythontech 0:5868e8752d44 799 // need to match a token
pythontech 0:5868e8752d44 800 mp_token_kind_t tok_kind = rule->arg[i] & RULE_ARG_ARG_MASK;
pythontech 0:5868e8752d44 801 if (lex->tok_kind == tok_kind) {
pythontech 0:5868e8752d44 802 // matched token
pythontech 0:5868e8752d44 803 if (tok_kind == MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 804 push_result_token(&parser);
pythontech 0:5868e8752d44 805 }
pythontech 0:5868e8752d44 806 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 807 } else {
pythontech 0:5868e8752d44 808 // failed to match token
pythontech 0:5868e8752d44 809 if (i > 0) {
pythontech 0:5868e8752d44 810 // already eaten tokens so can't backtrack
pythontech 0:5868e8752d44 811 goto syntax_error;
pythontech 0:5868e8752d44 812 } else {
pythontech 0:5868e8752d44 813 // this rule failed, so backtrack
pythontech 0:5868e8752d44 814 backtrack = true;
pythontech 0:5868e8752d44 815 goto next_rule;
pythontech 0:5868e8752d44 816 }
pythontech 0:5868e8752d44 817 }
pythontech 0:5868e8752d44 818 break;
pythontech 0:5868e8752d44 819 }
pythontech 0:5868e8752d44 820 case RULE_ARG_RULE:
pythontech 0:5868e8752d44 821 case RULE_ARG_OPT_RULE:
pythontech 0:5868e8752d44 822 rule_and_no_other_choice:
pythontech 0:5868e8752d44 823 push_rule(&parser, rule_src_line, rule, i + 1); // save this and-rule
pythontech 0:5868e8752d44 824 push_rule_from_arg(&parser, rule->arg[i]); // push child of and-rule
pythontech 0:5868e8752d44 825 goto next_rule;
pythontech 0:5868e8752d44 826 default:
pythontech 0:5868e8752d44 827 assert(0);
pythontech 0:5868e8752d44 828 goto rule_and_no_other_choice; // to help flow control analysis
pythontech 0:5868e8752d44 829 }
pythontech 0:5868e8752d44 830 }
pythontech 0:5868e8752d44 831
pythontech 0:5868e8752d44 832 assert(i == n);
pythontech 0:5868e8752d44 833
pythontech 0:5868e8752d44 834 // matched the rule, so now build the corresponding parse_node
pythontech 0:5868e8752d44 835
pythontech 0:5868e8752d44 836 // count number of arguments for the parse_node
pythontech 0:5868e8752d44 837 i = 0;
pythontech 0:5868e8752d44 838 bool emit_rule = false;
pythontech 0:5868e8752d44 839 for (size_t x = 0; x < n; ++x) {
pythontech 0:5868e8752d44 840 if ((rule->arg[x] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) {
pythontech 0:5868e8752d44 841 mp_token_kind_t tok_kind = rule->arg[x] & RULE_ARG_ARG_MASK;
pythontech 0:5868e8752d44 842 if (tok_kind >= MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 843 emit_rule = true;
pythontech 0:5868e8752d44 844 }
pythontech 0:5868e8752d44 845 if (tok_kind == MP_TOKEN_NAME) {
pythontech 0:5868e8752d44 846 // only tokens which were names are pushed to stack
pythontech 0:5868e8752d44 847 i += 1;
pythontech 0:5868e8752d44 848 }
pythontech 0:5868e8752d44 849 } else {
pythontech 0:5868e8752d44 850 // rules are always pushed
pythontech 0:5868e8752d44 851 i += 1;
pythontech 0:5868e8752d44 852 }
pythontech 0:5868e8752d44 853 }
pythontech 0:5868e8752d44 854
pythontech 0:5868e8752d44 855 #if !MICROPY_ENABLE_DOC_STRING
pythontech 0:5868e8752d44 856 // this code discards lonely statements, such as doc strings
pythontech 0:5868e8752d44 857 if (input_kind != MP_PARSE_SINGLE_INPUT && rule->rule_id == RULE_expr_stmt && peek_result(&parser, 0) == MP_PARSE_NODE_NULL) {
pythontech 0:5868e8752d44 858 mp_parse_node_t p = peek_result(&parser, 1);
pythontech 0:5868e8752d44 859 if ((MP_PARSE_NODE_IS_LEAF(p) && !MP_PARSE_NODE_IS_ID(p)) || MP_PARSE_NODE_IS_STRUCT_KIND(p, RULE_string)) {
pythontech 0:5868e8752d44 860 pop_result(&parser); // MP_PARSE_NODE_NULL
pythontech 0:5868e8752d44 861 mp_parse_node_t pn = pop_result(&parser); // possibly RULE_string
pythontech 0:5868e8752d44 862 if (MP_PARSE_NODE_IS_STRUCT(pn)) {
pythontech 0:5868e8752d44 863 mp_parse_node_struct_t *pns = (mp_parse_node_struct_t *)pn;
pythontech 0:5868e8752d44 864 if (MP_PARSE_NODE_STRUCT_KIND(pns) == RULE_string) {
pythontech 0:5868e8752d44 865 m_del(char, (char*)pns->nodes[0], (size_t)pns->nodes[1]);
pythontech 0:5868e8752d44 866 }
pythontech 0:5868e8752d44 867 }
pythontech 0:5868e8752d44 868 push_result_rule(&parser, rule_src_line, rules[RULE_pass_stmt], 0);
pythontech 0:5868e8752d44 869 break;
pythontech 0:5868e8752d44 870 }
pythontech 0:5868e8752d44 871 }
pythontech 0:5868e8752d44 872 #endif
pythontech 0:5868e8752d44 873
pythontech 0:5868e8752d44 874 // always emit these rules, even if they have only 1 argument
pythontech 0:5868e8752d44 875 if (rule->rule_id == RULE_expr_stmt || rule->rule_id == RULE_yield_stmt) {
pythontech 0:5868e8752d44 876 emit_rule = true;
pythontech 0:5868e8752d44 877 }
pythontech 0:5868e8752d44 878
pythontech 0:5868e8752d44 879 // if a rule has the RULE_ACT_ALLOW_IDENT bit set then this
pythontech 0:5868e8752d44 880 // rule should not be emitted if it has only 1 argument
pythontech 0:5868e8752d44 881 if (rule->act & RULE_ACT_ALLOW_IDENT) {
pythontech 0:5868e8752d44 882 emit_rule = false;
pythontech 0:5868e8752d44 883 }
pythontech 0:5868e8752d44 884
pythontech 0:5868e8752d44 885 // always emit these rules, and add an extra blank node at the end (to be used by the compiler to store data)
pythontech 0:5868e8752d44 886 if (ADD_BLANK_NODE(rule)) {
pythontech 0:5868e8752d44 887 emit_rule = true;
pythontech 0:5868e8752d44 888 push_result_node(&parser, MP_PARSE_NODE_NULL);
pythontech 0:5868e8752d44 889 i += 1;
pythontech 0:5868e8752d44 890 }
pythontech 0:5868e8752d44 891
pythontech 0:5868e8752d44 892 size_t num_not_nil = 0;
pythontech 0:5868e8752d44 893 for (size_t x = 0; x < i; ++x) {
pythontech 0:5868e8752d44 894 if (peek_result(&parser, x) != MP_PARSE_NODE_NULL) {
pythontech 0:5868e8752d44 895 num_not_nil += 1;
pythontech 0:5868e8752d44 896 }
pythontech 0:5868e8752d44 897 }
pythontech 0:5868e8752d44 898 if (emit_rule || num_not_nil != 1) {
pythontech 0:5868e8752d44 899 // need to add rule when num_not_nil==0 for, eg, atom_paren, testlist_comp_3b
pythontech 0:5868e8752d44 900 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 901 } else {
pythontech 0:5868e8752d44 902 // single result, leave it on stack
pythontech 0:5868e8752d44 903 mp_parse_node_t pn = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 904 for (size_t x = 0; x < i; ++x) {
pythontech 0:5868e8752d44 905 mp_parse_node_t pn2 = pop_result(&parser);
pythontech 0:5868e8752d44 906 if (pn2 != MP_PARSE_NODE_NULL) {
pythontech 0:5868e8752d44 907 pn = pn2;
pythontech 0:5868e8752d44 908 }
pythontech 0:5868e8752d44 909 }
pythontech 0:5868e8752d44 910 push_result_node(&parser, pn);
pythontech 0:5868e8752d44 911 }
pythontech 0:5868e8752d44 912 break;
pythontech 0:5868e8752d44 913 }
pythontech 0:5868e8752d44 914
pythontech 0:5868e8752d44 915 case RULE_ACT_LIST: {
pythontech 0:5868e8752d44 916 // n=2 is: item item*
pythontech 0:5868e8752d44 917 // n=1 is: item (sep item)*
pythontech 0:5868e8752d44 918 // n=3 is: item (sep item)* [sep]
pythontech 0:5868e8752d44 919 bool had_trailing_sep;
pythontech 0:5868e8752d44 920 if (backtrack) {
pythontech 0:5868e8752d44 921 list_backtrack:
pythontech 0:5868e8752d44 922 had_trailing_sep = false;
pythontech 0:5868e8752d44 923 if (n == 2) {
pythontech 0:5868e8752d44 924 if (i == 1) {
pythontech 0:5868e8752d44 925 // fail on item, first time round; propagate backtrack
pythontech 0:5868e8752d44 926 goto next_rule;
pythontech 0:5868e8752d44 927 } else {
pythontech 0:5868e8752d44 928 // fail on item, in later rounds; finish with this rule
pythontech 0:5868e8752d44 929 backtrack = false;
pythontech 0:5868e8752d44 930 }
pythontech 0:5868e8752d44 931 } else {
pythontech 0:5868e8752d44 932 if (i == 1) {
pythontech 0:5868e8752d44 933 // fail on item, first time round; propagate backtrack
pythontech 0:5868e8752d44 934 goto next_rule;
pythontech 0:5868e8752d44 935 } else if ((i & 1) == 1) {
pythontech 0:5868e8752d44 936 // fail on item, in later rounds; have eaten tokens so can't backtrack
pythontech 0:5868e8752d44 937 if (n == 3) {
pythontech 0:5868e8752d44 938 // list allows trailing separator; finish parsing list
pythontech 0:5868e8752d44 939 had_trailing_sep = true;
pythontech 0:5868e8752d44 940 backtrack = false;
pythontech 0:5868e8752d44 941 } else {
pythontech 0:5868e8752d44 942 // list doesn't allowing trailing separator; fail
pythontech 0:5868e8752d44 943 goto syntax_error;
pythontech 0:5868e8752d44 944 }
pythontech 0:5868e8752d44 945 } else {
pythontech 0:5868e8752d44 946 // fail on separator; finish parsing list
pythontech 0:5868e8752d44 947 backtrack = false;
pythontech 0:5868e8752d44 948 }
pythontech 0:5868e8752d44 949 }
pythontech 0:5868e8752d44 950 } else {
pythontech 0:5868e8752d44 951 for (;;) {
pythontech 0:5868e8752d44 952 size_t arg = rule->arg[i & 1 & n];
pythontech 0:5868e8752d44 953 switch (arg & RULE_ARG_KIND_MASK) {
pythontech 0:5868e8752d44 954 case RULE_ARG_TOK:
pythontech 0:5868e8752d44 955 if (lex->tok_kind == (arg & RULE_ARG_ARG_MASK)) {
pythontech 0:5868e8752d44 956 if (i & 1 & n) {
pythontech 0:5868e8752d44 957 // separators which are tokens are not pushed to result stack
pythontech 0:5868e8752d44 958 } else {
pythontech 0:5868e8752d44 959 push_result_token(&parser);
pythontech 0:5868e8752d44 960 }
pythontech 0:5868e8752d44 961 mp_lexer_to_next(lex);
pythontech 0:5868e8752d44 962 // got element of list, so continue parsing list
pythontech 0:5868e8752d44 963 i += 1;
pythontech 0:5868e8752d44 964 } else {
pythontech 0:5868e8752d44 965 // couldn't get element of list
pythontech 0:5868e8752d44 966 i += 1;
pythontech 0:5868e8752d44 967 backtrack = true;
pythontech 0:5868e8752d44 968 goto list_backtrack;
pythontech 0:5868e8752d44 969 }
pythontech 0:5868e8752d44 970 break;
pythontech 0:5868e8752d44 971 case RULE_ARG_RULE:
pythontech 0:5868e8752d44 972 rule_list_no_other_choice:
pythontech 0:5868e8752d44 973 push_rule(&parser, rule_src_line, rule, i + 1); // save this list-rule
pythontech 0:5868e8752d44 974 push_rule_from_arg(&parser, arg); // push child of list-rule
pythontech 0:5868e8752d44 975 goto next_rule;
pythontech 0:5868e8752d44 976 default:
pythontech 0:5868e8752d44 977 assert(0);
pythontech 0:5868e8752d44 978 goto rule_list_no_other_choice; // to help flow control analysis
pythontech 0:5868e8752d44 979 }
pythontech 0:5868e8752d44 980 }
pythontech 0:5868e8752d44 981 }
pythontech 0:5868e8752d44 982 assert(i >= 1);
pythontech 0:5868e8752d44 983
pythontech 0:5868e8752d44 984 // compute number of elements in list, result in i
pythontech 0:5868e8752d44 985 i -= 1;
pythontech 0:5868e8752d44 986 if ((n & 1) && (rule->arg[1] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) {
pythontech 0:5868e8752d44 987 // don't count separators when they are tokens
pythontech 0:5868e8752d44 988 i = (i + 1) / 2;
pythontech 0:5868e8752d44 989 }
pythontech 0:5868e8752d44 990
pythontech 0:5868e8752d44 991 if (i == 1) {
pythontech 0:5868e8752d44 992 // list matched single item
pythontech 0:5868e8752d44 993 if (had_trailing_sep) {
pythontech 0:5868e8752d44 994 // if there was a trailing separator, make a list of a single item
pythontech 0:5868e8752d44 995 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 996 } else {
pythontech 0:5868e8752d44 997 // just leave single item on stack (ie don't wrap in a list)
pythontech 0:5868e8752d44 998 }
pythontech 0:5868e8752d44 999 } else {
pythontech 0:5868e8752d44 1000 push_result_rule(&parser, rule_src_line, rule, i);
pythontech 0:5868e8752d44 1001 }
pythontech 0:5868e8752d44 1002 break;
pythontech 0:5868e8752d44 1003 }
pythontech 0:5868e8752d44 1004
pythontech 0:5868e8752d44 1005 default:
pythontech 0:5868e8752d44 1006 assert(0);
pythontech 0:5868e8752d44 1007 }
pythontech 0:5868e8752d44 1008 }
pythontech 0:5868e8752d44 1009
pythontech 0:5868e8752d44 1010 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 1011 mp_map_deinit(&parser.consts);
pythontech 0:5868e8752d44 1012 #endif
pythontech 0:5868e8752d44 1013
pythontech 0:5868e8752d44 1014 // truncate final chunk and link into chain of chunks
pythontech 0:5868e8752d44 1015 if (parser.cur_chunk != NULL) {
pythontech 0:5868e8752d44 1016 (void)m_renew(byte, parser.cur_chunk,
pythontech 0:5868e8752d44 1017 sizeof(mp_parse_chunk_t) + parser.cur_chunk->alloc,
pythontech 0:5868e8752d44 1018 sizeof(mp_parse_chunk_t) + parser.cur_chunk->union_.used);
pythontech 0:5868e8752d44 1019 parser.cur_chunk->alloc = parser.cur_chunk->union_.used;
pythontech 0:5868e8752d44 1020 parser.cur_chunk->union_.next = parser.tree.chunk;
pythontech 0:5868e8752d44 1021 parser.tree.chunk = parser.cur_chunk;
pythontech 0:5868e8752d44 1022 }
pythontech 0:5868e8752d44 1023
pythontech 0:5868e8752d44 1024 mp_obj_t exc;
pythontech 0:5868e8752d44 1025
pythontech 0:5868e8752d44 1026 if (parser.parse_error) {
pythontech 0:5868e8752d44 1027 #if MICROPY_COMP_CONST
pythontech 0:5868e8752d44 1028 if (parser.parse_error == PARSE_ERROR_CONST) {
pythontech 0:5868e8752d44 1029 exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
pythontech 0:5868e8752d44 1030 "constant must be an integer");
pythontech 0:5868e8752d44 1031 } else
pythontech 0:5868e8752d44 1032 #endif
pythontech 0:5868e8752d44 1033 {
pythontech 0:5868e8752d44 1034 assert(parser.parse_error == PARSE_ERROR_MEMORY);
pythontech 0:5868e8752d44 1035 memory_error:
pythontech 0:5868e8752d44 1036 exc = mp_obj_new_exception_msg(&mp_type_MemoryError,
pythontech 0:5868e8752d44 1037 "parser could not allocate enough memory");
pythontech 0:5868e8752d44 1038 }
pythontech 0:5868e8752d44 1039 parser.tree.root = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 1040 } else if (
pythontech 0:5868e8752d44 1041 lex->tok_kind != MP_TOKEN_END // check we are at the end of the token stream
pythontech 0:5868e8752d44 1042 || parser.result_stack_top == 0 // check that we got a node (can fail on empty input)
pythontech 0:5868e8752d44 1043 ) {
pythontech 0:5868e8752d44 1044 syntax_error:
pythontech 0:5868e8752d44 1045 if (lex->tok_kind == MP_TOKEN_INDENT) {
pythontech 0:5868e8752d44 1046 exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
pythontech 0:5868e8752d44 1047 "unexpected indent");
pythontech 0:5868e8752d44 1048 } else if (lex->tok_kind == MP_TOKEN_DEDENT_MISMATCH) {
pythontech 0:5868e8752d44 1049 exc = mp_obj_new_exception_msg(&mp_type_IndentationError,
pythontech 0:5868e8752d44 1050 "unindent does not match any outer indentation level");
pythontech 0:5868e8752d44 1051 } else {
pythontech 0:5868e8752d44 1052 exc = mp_obj_new_exception_msg(&mp_type_SyntaxError,
pythontech 0:5868e8752d44 1053 "invalid syntax");
pythontech 0:5868e8752d44 1054 }
pythontech 0:5868e8752d44 1055 parser.tree.root = MP_PARSE_NODE_NULL;
pythontech 0:5868e8752d44 1056 } else {
pythontech 0:5868e8752d44 1057 // no errors
pythontech 0:5868e8752d44 1058
pythontech 0:5868e8752d44 1059 //result_stack_show(parser);
pythontech 0:5868e8752d44 1060 //printf("rule stack alloc: %d\n", parser.rule_stack_alloc);
pythontech 0:5868e8752d44 1061 //printf("result stack alloc: %d\n", parser.result_stack_alloc);
pythontech 0:5868e8752d44 1062 //printf("number of parse nodes allocated: %d\n", num_parse_nodes_allocated);
pythontech 0:5868e8752d44 1063
pythontech 0:5868e8752d44 1064 // get the root parse node that we created
pythontech 0:5868e8752d44 1065 assert(parser.result_stack_top == 1);
pythontech 0:5868e8752d44 1066 exc = MP_OBJ_NULL;
pythontech 0:5868e8752d44 1067 parser.tree.root = parser.result_stack[0];
pythontech 0:5868e8752d44 1068 }
pythontech 0:5868e8752d44 1069
pythontech 0:5868e8752d44 1070 // free the memory that we don't need anymore
pythontech 0:5868e8752d44 1071 m_del(rule_stack_t, parser.rule_stack, parser.rule_stack_alloc);
pythontech 0:5868e8752d44 1072 m_del(mp_parse_node_t, parser.result_stack, parser.result_stack_alloc);
pythontech 0:5868e8752d44 1073 // we also free the lexer on behalf of the caller (see below)
pythontech 0:5868e8752d44 1074
pythontech 0:5868e8752d44 1075 if (exc != MP_OBJ_NULL) {
pythontech 0:5868e8752d44 1076 // had an error so raise the exception
pythontech 0:5868e8752d44 1077 // add traceback to give info about file name and location
pythontech 0:5868e8752d44 1078 // we don't have a 'block' name, so just pass the NULL qstr to indicate this
pythontech 0:5868e8752d44 1079 mp_obj_exception_add_traceback(exc, lex->source_name, lex->tok_line, MP_QSTR_NULL);
pythontech 0:5868e8752d44 1080 mp_lexer_free(lex);
pythontech 0:5868e8752d44 1081 nlr_raise(exc);
pythontech 0:5868e8752d44 1082 } else {
pythontech 0:5868e8752d44 1083 mp_lexer_free(lex);
pythontech 0:5868e8752d44 1084 return parser.tree;
pythontech 0:5868e8752d44 1085 }
pythontech 0:5868e8752d44 1086 }
pythontech 0:5868e8752d44 1087
pythontech 0:5868e8752d44 1088 void mp_parse_tree_clear(mp_parse_tree_t *tree) {
pythontech 0:5868e8752d44 1089 mp_parse_chunk_t *chunk = tree->chunk;
pythontech 0:5868e8752d44 1090 while (chunk != NULL) {
pythontech 0:5868e8752d44 1091 mp_parse_chunk_t *next = chunk->union_.next;
pythontech 0:5868e8752d44 1092 m_del(byte, chunk, sizeof(mp_parse_chunk_t) + chunk->alloc);
pythontech 0:5868e8752d44 1093 chunk = next;
pythontech 0:5868e8752d44 1094 }
pythontech 0:5868e8752d44 1095 }
pythontech 0:5868e8752d44 1096
pythontech 0:5868e8752d44 1097 #endif // MICROPY_ENABLE_COMPILER