USB device stack, with KL25Z fixes for USB 3.0 hosts and sleep/resume interrupt handling

Dependents:   frdm_Slider_Keyboard idd_hw2_figlax_PanType idd_hw2_appachu_finger_chording idd_hw3_AngieWangAntonioDeLimaFernandesDanielLim_BladeSymphony ... more

Fork of USBDevice by mbed official

This is an overhauled version of the standard mbed USB device-side driver library, with bug fixes for KL25Z devices. It greatly improves reliability and stability of USB on the KL25Z, especially with devices using multiple endpoints concurrently.

I've had some nagging problems with the base mbed implementation for a long time, manifesting as occasional random disconnects that required rebooting the device. Recently (late 2015), I started implementing a USB device on the KL25Z that used multiple endpoints, and suddenly the nagging, occasional problems turned into frequent and predictable crashes. This forced me to delve into the USB stack and figure out what was really going on. Happily, the frequent crashes made it possible to track down and fix the problems. This new version is working very reliably in my testing - the random disconnects seem completely eradicated, even under very stressful conditions for the device.

Summary

  • Overall stability improvements
  • USB 3.0 host support
  • Stalled endpoint fixes
  • Sleep/resume notifications
  • Smaller memory footprint
  • General code cleanup

Update - 2/15/2016

My recent fixes introduced a new problem that made the initial connection fail most of the time on certain hosts. It's not clear if the common thread was a particular type of motherboard or USB chip set, or a specific version of Windows, or what, but several people ran into it. We tracked the problem down to the "stall" fixes in the earlier updates, which we now know weren't quite the right fixes after all. The latest update (2/15/2016) fixes this. It has new and improved "unstall" handling that so far works well with diverse hosts.

Race conditions and overall stability

The base mbed KL25Z implementation has a lot of problems with "race conditions" - timing problems that can happen when hardware interrupts occur at inopportune moments. The library shares a bunch of static variable data between interrupt handler context and regular application context. This isn't automatically a bad thing, but it does require careful coordination to make sure that the interrupt handler doesn't corrupt data that the other code was in the middle of updating when an interrupt occurs. The base mbed code, though, doesn't do any of the necessary coordination. This makes it kind of amazing that the base code worked at all for anyone, but I guess the interrupt rate is low enough in most applications that the glitch rate was below anyone's threshold to seriously investigate.

This overhaul adds the necessary coordination for the interrupt handlers to protect against these data corruptions. I think it's very solid now, and hopefully entirely free of the numerous race conditions in the old code. It's always hard to be certain that you've fixed every possible bug like this because they strike (effectively) at random, but I'm pretty confident: my test application was reliably able to trigger glitches in the base code in a matter of minutes, but the same application (with the overhauled library) now runs for days on end without dropping the connection.

Stalled endpoint fixes

USB has a standard way of handling communications errors called a "stall", which basically puts the connection into an error mode to let both sides know that they need to reset their internal states and sync up again. The original mbed version of the USB device library doesn't seem to have the necessary code to recover from this condition properly. The KL25Z hardware does some of the work, but it also seems to require the software to take some steps to "un-stall" the connection. (I keep saying "seems to" because the hardware reference material is very sketchy about all of this. Most of what I've figured out is from observing the device in action with a Windows host.) This new version adds code to do the necessary re-syncing and get the connection going again, automatically, and transparently to the user.

USB 3.0 Hosts

The original mbed code sometimes didn't work when connecting to hosts with USB 3.0 ports. This didn't affect every host, but it affected many of them. The common element seemed to be the Intel Haswell chip set on the host, but there may be other chip sets affected as well. In any case, the problem affected many PCs from the Windows 7 and 8 generation, as well as many Macs. It was possible to work around the problem by avoiding USB 3.0 ports - you could use a USB 2 port on the host, or plug a USB 2 hub between the host and device. But I wanted to just fix the problem and eliminate the need for such workarounds. This modified version of the library has such a fix, which so far has worked for everyone who's tried.

Sleep/resume notifications

This modified version also contains an innocuous change to the KL25Z USB HAL code to handle sleep and resume interrupts with calls to suspendStateChanged(). The original KL25Z code omitted these calls (and in fact didn't even enable the interrupts), but I think this was an unintentional oversight - the notifier function is part of the generic API, and other supported boards all implement it. I use this feature in my own application so that I can distinguish sleep mode from actual disconnects and handle the two conditions correctly.

Smaller memory footprint

The base mbed version of the code allocates twice as much memory for USB buffers as it really needed to. It looks like the original developers intended to implement the KL25Z USB hardware's built-in double-buffering mechanism, but they ultimately abandoned that effort. But they left in the double memory allocation. This version removes that and allocates only what's actually needed. The USB buffers aren't that big (128 bytes per endpoint), so this doesn't save a ton of memory, but even a little memory is pretty precious on this machine given that it only has 16K.

(I did look into adding the double-buffering support that the original developers abandoned, but after some experimentation I decided they were right to skip it. It just doesn't seem to mesh well with the design of the rest of the mbed USB code. I think it would take a major rewrite to make it work, and it doesn't seem worth the effort given that most applications don't need it - it would only benefit applications that are moving so much data through USB that they're pushing the limits of the CPU. And even for those, I think it would be a lot simpler to build a purely software-based buffer rotation mechanism.)

General code cleanup

The KL25Z HAL code in this version has greatly expanded commentary and a lot of general cleanup. Some of the hardware constants were given the wrong symbolic names (e.g., EVEN and ODD were reversed), and many were just missing (written as hard-coded numbers without explanation). I fixed the misnomers and added symbolic names for formerly anonymous numbers. Hopefully the next person who has to overhaul this code will at least have an easier time understanding what I thought I was doing!

USBDevice/USBHAL_KL25Z.cpp

Committer:
mjr
Date:
2016-02-11
Revision:
47:c91e6d7762e4
Parent:
40:cd877d5c09ea

File content as of revision 47:c91e6d7762e4:

/* Copyright (c) 2010-2011 mbed.org, MIT License
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of this software
* and associated documentation files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all copies or
* substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
* BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
* DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

#if defined(TARGET_KL25Z) | defined(TARGET_KL46Z) | defined(TARGET_K20D5M) | defined(TARGET_K64F)

//#define DEBUG
#ifdef DEBUG
#define printd(fmt, ...) printf(fmt, __VA_ARGS__)
#else
#define printd(fmt, ...)
#endif


#include "USBHAL.h"

// Critical section controls.  This module uses a bunch of static variables,
// and much of the code that accesses the statics can be called from either
// normal application context or IRQ context.  Whenever a shared variable is
// accessed from code that can run in an application context, we have to
// protect against interrupts by entering a critical section.  These macros
// enable and disable the USB IRQ if we're running in application context.
// (They do nothing if we're already in interrupt context, because the
// hardware interrupt controller won't generated another of the same IRQ
// that we're already handling.  We could still be interrupted by a different,
// higher-priority IRQ, but our shared variables are only shared within this
// module, so they won't be affected by other interrupt handlers.)
static int inIRQ;
#define ENTER_CRITICAL_SECTION \
    if (!inIRQ) \
        NVIC_DisableIRQ(USB0_IRQn);
#define EXIT_CRITICAL_SECTION \
    if (!inIRQ) \
        NVIC_EnableIRQ(USB0_IRQn);

// static singleton instance pointer
USBHAL * USBHAL::instance;

// Convert physical endpoint number to register bit
#define EP(endpoint) (1<<(endpoint))

// Convert physical endpoint number to logical endpoint number.
// Each logical endpoint has two physical endpoints, one RX and 
// one TX.  The physical endpoints are numbered in RX,TX pairs,
// so the logical endpoint number is simply the physical endpoint
// number divided by 2 (discarding the remainder).
#define PHY_TO_LOG(endpoint)    ((endpoint)>>1)

// Get a physical endpoint's direction.  IN and OUT are from
// the host's perspective, so from our perspective on the device,
// IN == TX and OUT == RX.  The physical endpoints are in RX,TX
// pairs, so the OUT/RX is the even numbered element of a pair
// and the IN/TX is the odd numbered element.
#define IN_EP(endpoint)     ((endpoint) & 1U ? true : false)
#define OUT_EP(endpoint)    ((endpoint) & 1U ? false : true)

// BDT status flags, defined by the SIE hardware.  These are
// bits packed into the 'info' byte of a BDT entry.
#define BD_OWN_MASK        (1<<7)       // OWN - hardware SIE owns the BDT (TX/RX in progress)
#define BD_DATA01_MASK     (1<<6)       // DATA01 - DATA0/DATA1 bit for current TX/RX on endpoint
#define BD_KEEP_MASK       (1<<5)       // KEEP - hardware keeps BDT ownership after token completes
#define BD_NINC_MASK       (1<<4)       // NO INCREMENT - buffer location is a FIFO, so use same address for all bytes
#define BD_DTS_MASK        (1<<3)       // DATA TOGGLE SENSING - hardware SIE checks for DATA0/DATA1 match during RX/TX
#define BD_STALL_MASK      (1<<2)       // STALL - SIE issues STALL handshake in reply to any host access to endpoint

// Endpoint direction
#define TX    1
#define RX    0

// Buffer parity.  The hardware has a double-buffering scheme where each
// physical endpoint has two associated BDT entries, labeled EVEN and ODD.
// We disable the double buffering, so only the EVEN buffers are used in
// this implementation.
#define EVEN  0
#define ODD   1

// Get the BDT index for a given logical endpoint, direction, and buffer parity
#define EP_BDT_IDX(logep, dir, odd) (((logep) * 4) + (2 * (dir)) + (1 *  (odd)))

// Get the BDT index for a given physical endpoint and buffer parity
#define PEP_BDT_IDX(phyep, odd)  (((phyep) * 2) + (1 * (odd)))

// Token types reported in the BDT 'info' flags.  
#define TOK_PID(idx)   ((bdt[idx].info >> 2) & 0x0F)
#define SETUP_TOKEN    0x0D
#define IN_TOKEN       0x09
#define OUT_TOKEN      0x01

// Buffer Descriptor Table (BDT) entry.  This is the hardware-defined
// memory structure for the shared memory block controlling an endpoint.
typedef struct BDT {
    uint8_t   info;       // BD[0:7]
    uint8_t   dummy;      // RSVD: BD[8:15]
    uint16_t  byte_count; // BD[16:32]
    uint32_t  address;    // Addr
} BDT;

// There are:
//    * 16 bidirectional logical endpoints -> 32 physical endpoints
//    * 2 BDT entries per endpoint (EVEN/ODD) -> 64 BDT entries
__attribute__((__aligned__(512))) BDT bdt[NUMBER_OF_PHYSICAL_ENDPOINTS * 2];

// Transfer buffers.  We allocate the transfer buffers and point the
// SIE hardware to them via the BDT.  We disable hardware SIE's
// double-buffering (EVEN/ODD) scheme, so we only allocate one buffer
// per physical endpoint.
uint8_t *endpoint_buffer[NUMBER_OF_PHYSICAL_ENDPOINTS];

// Allocated size of each endpoint buffer
size_t epMaxPacket[NUMBER_OF_PHYSICAL_ENDPOINTS];

// SET ADDRESS mode tracking.  The address assignment has to be done in a
// specific order and with specific timing defined by the USB setup protocol 
// standards.  To get the sequencing right, we set a flag when we get the
// address message, and then set the address in the SIE when we're at the 
// right subsequent packet step in the protocol exchange.  These variables
// are just a place to stash the information between the time we receive the
// data and the time we're ready to update the SIE register.
static uint8_t set_addr = 0;
static uint8_t addr = 0;

// Endpoint DATA0/DATA1 bits, packed as a bit vector.  Each endpoint's
// bit is at (1 << endpoint number).  These track the current bit value
// on the endpoint.  For TX endpoints, this is the bit for the LAST
// packet we sent (so the next packet will be the inverse).  For RX
// endpoints, this is the bit value we expect for the NEXT packet.
// (Yes, it's inconsistent.)
static uint32_t Data1  = 0x55555555;

// Endpoint read/write completion flags, packed as a bit vector.  Each 
// endpoint's bit is at (1 << endpoint number).  A 1 bit signifies that
// the last read or write has completed (and hasn't had its result 
// consumed yet).
static volatile int epComplete = 0;

static uint32_t frameNumber() 
{
    return((USB0->FRMNUML | (USB0->FRMNUMH << 8)) & 0x07FF);
}

uint32_t USBHAL::endpointReadcore(uint8_t endpoint, uint8_t *buffer) 
{
    return 0;
}

USBHAL::USBHAL(void) {
    // Disable IRQ
    NVIC_DisableIRQ(USB0_IRQn);

#if defined(TARGET_K64F)
    MPU->CESR=0;
#endif
    // fill in callback array
    epCallback[0] = &USBHAL::EP0_OUT_callback;
    epCallback[1] = &USBHAL::EP0_IN_callback;
    epCallback[2] = &USBHAL::EP1_OUT_callback;
    epCallback[3] = &USBHAL::EP1_IN_callback;
    epCallback[4] = &USBHAL::EP2_OUT_callback;
    epCallback[5] = &USBHAL::EP2_IN_callback;
    epCallback[6] = &USBHAL::EP3_OUT_callback;
    epCallback[7] = &USBHAL::EP3_IN_callback;
    epCallback[8] = &USBHAL::EP4_OUT_callback;
    epCallback[9] = &USBHAL::EP4_IN_callback;
    epCallback[10] = &USBHAL::EP5_OUT_callback;
    epCallback[11] = &USBHAL::EP5_IN_callback;
    epCallback[12] = &USBHAL::EP6_OUT_callback;
    epCallback[13] = &USBHAL::EP6_IN_callback;
    epCallback[14] = &USBHAL::EP7_OUT_callback;
    epCallback[15] = &USBHAL::EP7_IN_callback;
    epCallback[16] = &USBHAL::EP8_OUT_callback;
    epCallback[17] = &USBHAL::EP8_IN_callback;
    epCallback[18] = &USBHAL::EP9_OUT_callback;
    epCallback[19] = &USBHAL::EP9_IN_callback;
    epCallback[20] = &USBHAL::EP10_OUT_callback;
    epCallback[21] = &USBHAL::EP10_IN_callback;
    epCallback[22] = &USBHAL::EP11_OUT_callback;
    epCallback[23] = &USBHAL::EP11_IN_callback;
    epCallback[24] = &USBHAL::EP12_OUT_callback;
    epCallback[25] = &USBHAL::EP12_IN_callback;
    epCallback[26] = &USBHAL::EP13_OUT_callback;
    epCallback[27] = &USBHAL::EP13_IN_callback;
    epCallback[28] = &USBHAL::EP14_OUT_callback;
    epCallback[29] = &USBHAL::EP14_IN_callback;
    epCallback[30] = &USBHAL::EP15_OUT_callback;
    epCallback[31] = &USBHAL::EP15_IN_callback;

    // choose usb src as PLL
    SIM->SOPT2 |= (SIM_SOPT2_USBSRC_MASK | SIM_SOPT2_PLLFLLSEL_MASK);

    // enable OTG clock
    SIM->SCGC4 |= SIM_SCGC4_USBOTG_MASK;

    // Attach IRQ
    instance = this;
    NVIC_SetVector(USB0_IRQn, (uint32_t)&_usbisr);
    NVIC_EnableIRQ(USB0_IRQn);

    // USB Module Configuration
    // Reset USB Module
    USB0->USBTRC0 |= USB_USBTRC0_USBRESET_MASK;
    while(USB0->USBTRC0 & USB_USBTRC0_USBRESET_MASK);

    // Set BDT Base Register
    USB0->BDTPAGE1 = (uint8_t)((uint32_t)bdt>>8);
    USB0->BDTPAGE2 = (uint8_t)((uint32_t)bdt>>16);
    USB0->BDTPAGE3 = (uint8_t)((uint32_t)bdt>>24);

    // Clear interrupt flag
    USB0->ISTAT = 0xff;

    // USB Interrupt Enablers
    USB0->INTEN |= USB_INTEN_TOKDNEEN_MASK |
                   USB_INTEN_SOFTOKEN_MASK |
                   USB_INTEN_ERROREN_MASK  |
                   USB_INTEN_SLEEPEN_MASK |
                   USB_INTEN_RESUMEEN_MASK |
                   USB_INTEN_USBRSTEN_MASK;

    // Disable weak pull downs
    USB0->USBCTRL &= ~(USB_USBCTRL_PDE_MASK | USB_USBCTRL_SUSP_MASK);

    USB0->USBTRC0 |= 0x40;
}

USBHAL::~USBHAL(void) 
{ 
}

void USBHAL::connect(void) 
{
    // enable USB
    USB0->CTL |= USB_CTL_USBENSOFEN_MASK;

    // Pull up enable
    USB0->CONTROL |= USB_CONTROL_DPPULLUPNONOTG_MASK;
}

void USBHAL::disconnect(void) 
{
    // disable USB
    USB0->CTL &= ~USB_CTL_USBENSOFEN_MASK;

    // Pull up disable
    USB0->CONTROL &= ~USB_CONTROL_DPPULLUPNONOTG_MASK;

    //Free buffers if required:
    for (int i = 0 ; i < NUMBER_OF_PHYSICAL_ENDPOINTS ; i++) 
    {
        if (endpoint_buffer[i] != NULL)
        {
            free(endpoint_buffer[i]);
            endpoint_buffer[i] = NULL;
            epMaxPacket[i] = 0;
        }
    }
}

void USBHAL::configureDevice(void) 
{
    // not needed
}

void USBHAL::unconfigureDevice(void) 
{
    // not needed
}

void USBHAL::setAddress(uint8_t address) 
{
    // we don't set the address now otherwise the usb controller does not ack
    // we set a flag instead
    // see usbisr when an IN token is received
    set_addr = 1;
    addr = address;
 }

bool USBHAL::realiseEndpoint(uint8_t endpoint, uint32_t maxPacket, uint32_t flags) 
{
    // validate the endpoint number
    if (endpoint >= NUMBER_OF_PHYSICAL_ENDPOINTS)
        return false;

    // get the logical endpoint
    uint32_t log_endpoint = PHY_TO_LOG(endpoint);
    
    // Constrain the requested packet size to the maximum for the endpoint type.
    // Full Speed USB allows up to 1023 bytes for isochronous endpoints and 64 bytes
    // for bulk and interrupt endpoints.
    uint32_t realMaxPacket = ((flags & ISOCHRONOUS) ? 1023 : 64);
    if (maxPacket > realMaxPacket)
        maxPacket = realMaxPacket;

    // Use the HANDSHAKE flag for non-isochronous endpoints.  Don't use handshaking
    // for an iso endpoint, since this type of endpoint is for applications like
    // audio and video streaming where it's preferable to ignore lost packets and
    // just carry on with the latest data.
    uint32_t ctlFlags = 0;
    if (!(flags & ISOCHRONOUS))
        ctlFlags |= USB_ENDPT_EPHSHK_MASK;
        
    // figure the RX/TX based on the endpoint direction
    ctlFlags |= (IN_EP(endpoint) ? USB_ENDPT_EPTXEN_MASK : USB_ENDPT_EPRXEN_MASK);

    ENTER_CRITICAL_SECTION
    {
        // if we don't already have a buffer that's big enough, allocate a new one
        uint8_t *buf = endpoint_buffer[endpoint];
        if (buf == 0 || maxPacket > epMaxPacket[endpoint])
        {
            // free any existing buffer
            if (buf != 0)
                free(buf);
                
            // allocate a new one
            epMaxPacket[endpoint] = maxPacket;
            endpoint_buffer[endpoint] = buf = (uint8_t *)malloc(maxPacket);
        }
        
        // Set up the BDT entry.  Note that we disable the hardware double-buffering
        // scheme, so we only use the EVEN buffer for the endpoint.
        int idx = PEP_BDT_IDX(endpoint, EVEN);
        bdt[idx].info = 0;
        bdt[idx].address = (uint32_t)buf;
        
        // Set the endpoint flags.  Note that these bits are additive, since the
        // endpoint register represents the logical endpoint, which is the combination
        // of the physical IN and OUT endpoints.
        USB0->ENDPOINT[log_endpoint].ENDPT |= ctlFlags;
    
        // If this is an OUT endpoint, queue the first read on the endpoint by
        // handing ownership of the BDT to the SIE.
        if (OUT_EP(endpoint))
        {
            bdt[idx].byte_count = maxPacket;
            bdt[idx].info = BD_OWN_MASK | BD_DTS_MASK;
        }
    
        // Set DATA1 on the endpoint.  For RX endpoints, we just queued up our first
        // read, which will always be a DATA0 packet, so the next read will use DATA1.
        // For TX endpoints, we always flip the bit *before* sending the packet, so
        // (counterintuitively) we need to set the DATA1 bit now to send DATA0 in the
        // next packet.  So in either case, we want DATA1 initially.
        Data1 |= (1 << endpoint);
    }
    EXIT_CRITICAL_SECTION
    
    // success
    return true;
}

// read setup packet
void USBHAL::EP0setup(uint8_t *buffer) 
{
    uint32_t sz;
    endpointReadResult(EP0OUT, buffer, &sz);
}

void USBHAL::EP0readStage(void) 
{
    // set DATA0 for the next packet
    Data1 &= ~1UL;
    
    // if we haven't already, give the BDT to the SIE to read the packet
    if (!(bdt[0].info & BD_OWN_MASK))
        bdt[0].info = (BD_DTS_MASK | BD_OWN_MASK);
}

void USBHAL::EP0read(void) 
{
    if (!(bdt[0].info & BD_OWN_MASK))
        bdt[0].byte_count = MAX_PACKET_SIZE_EP0;
}

uint32_t USBHAL::EP0getReadResult(uint8_t *buffer) 
{
    uint32_t sz;
    endpointReadResult(EP0OUT, buffer, &sz);
    return sz;
}

void USBHAL::EP0write(uint8_t *buffer, uint32_t size) 
{
    endpointWrite(EP0IN, buffer, size);
}

void USBHAL::EP0getWriteResult(void) 
{
}

void USBHAL::EP0stall(void) 
{
    printd("EP0 stall!\r\n");
    stallEndpoint(EP0OUT);
}

EP_STATUS USBHAL::endpointRead(uint8_t endpoint, uint32_t maximumSize) 
{
    // We always start a new read when we fetch the result of the
    // previous read, so we don't have to do anything here.  Simply
    // indicate that the read is pending so that the caller can proceed
    // to check the results.
    return EP_PENDING;
}

EP_STATUS USBHAL::endpointReadResult(uint8_t endpoint, uint8_t *buffer, uint32_t *bytesRead) 
{
    // validate the endpoint number and direction
    if (endpoint >= NUMBER_OF_PHYSICAL_ENDPOINTS || !OUT_EP(endpoint))
        return EP_INVALID;

    // get the logical endpoint
    uint32_t log_endpoint = PHY_TO_LOG(endpoint);
    
    // get the mode - it's isochronous if it doesn't have the handshake flag
    bool iso = !(USB0->ENDPOINT[log_endpoint].ENDPT & USB_ENDPT_EPHSHK_MASK);
    
    // get the BDT index
    int idx = EP_BDT_IDX(log_endpoint, RX, 0);
        
    // If the "complete" flag isn't set, the read is still pending in the SIE.
    // This doesn't apply the isochronous endpoints, since we don't get TOKDNE
    // interrupts on those (we use the SOF signal instead).  It also doesn't
    // apply to endpoint 0, since that doesn't use the epComplete mechanism
    // at all (necessary because we handle all transactions on this endpoint
    // in IRQ context).  For EP0, just make sure the hardware doesn't still
    // own the BDT - if it does, the last read hasn't completed yet.
    if (log_endpoint == 0)
    {
        // control endpoint - just make sure we own the BDT
 //$$$       if (bdt[idx].info & BD_OWN_MASK)
 //$$$           return EP_PENDING;
    }
    else if (!iso && !(epComplete & EP(endpoint)))
        return EP_PENDING;
    
    // get the buffer
    uint8_t *ep_buf = endpoint_buffer[endpoint];

    ENTER_CRITICAL_SECTION
    {
        // get the packet size from the BDT
        uint32_t sz  = bdt[idx].byte_count;
    
        // note if it's a SETUP token    
        bool setup = (log_endpoint == 0 && TOK_PID(idx) == SETUP_TOKEN);
    
        // copy the data
        memcpy(buffer, ep_buf, sz);
        *bytesRead = sz;

        // Figure the DATA0/DATA1 bit for the next packet received on this
        // endpoint.  The bit normally toggles on each packet, but it's
        // special for SETUP packets on endpoint 0.  The next OUT packet
        // after a SETUP packet with no data stage is always DATA0, even
        // if the SETUP packet was also DATA0.
        if (((Data1 >> endpoint) & 1) == ((bdt[idx].info >> 6) & 1)) 
        {
            if (setup && (buffer[6] == 0))  // if no setup data stage,
                Data1 &= ~1UL;              // set DATA0
            else
                Data1 ^= (1 << endpoint);   // otherwise just toggle the last bit
        }

        // set up the BDT entry to receive the next packet, and hand it to the SIE to fill    
        bdt[idx].byte_count = epMaxPacket[endpoint];
        bdt[idx].info = BD_DTS_MASK | BD_OWN_MASK | (((Data1 >> endpoint) & 1) << 6);
    
        // clear the SUSPEND TOKEN BUSY flag to allow the SIE to continue processing tokens
        USB0->CTL &= ~USB_CTL_TXSUSPENDTOKENBUSY_MASK;
    
        // clear the completion flag
        epComplete &= ~EP(endpoint);
    }
    EXIT_CRITICAL_SECTION
        
    // the read is completed
    return EP_COMPLETED;
}

EP_STATUS USBHAL::endpointWrite(uint8_t endpoint, uint8_t *data, uint32_t size) 
{
    // validate the endpoint number and direction
    if (endpoint >= NUMBER_OF_PHYSICAL_ENDPOINTS || !IN_EP(endpoint))
        return EP_INVALID;

    // get the BDT index
    int idx = EP_BDT_IDX(PHY_TO_LOG(endpoint), TX, 0);
    
    // get the buffer pointer
    uint8_t *buf = endpoint_buffer[endpoint];

    ENTER_CRITICAL_SECTION
    {
        // copy the data to the hardware buffer
        bdt[idx].byte_count = size;
        memcpy(buf, data, size);

        // flip the DATA1 bit before sending    
        Data1 ^= (1 << endpoint);

        // hand the BDT to the SIE hardware, and set the current DATA1 bit
        bdt[idx].info = BD_OWN_MASK | BD_DTS_MASK | (((Data1 >> endpoint) & 1) << 6);
    }
    EXIT_CRITICAL_SECTION

    // the operation is now pending
    return EP_PENDING;
}

EP_STATUS USBHAL::endpointWriteResult(uint8_t endpoint) 
{
    EP_STATUS result = EP_PENDING;

    ENTER_CRITICAL_SECTION
    {
        if (epComplete & EP(endpoint)) {
            epComplete &= ~EP(endpoint);
            result = EP_COMPLETED;
        }
    }
    EXIT_CRITICAL_SECTION
    
    return result;
}

void USBHAL::stallEndpoint(uint8_t endpoint) 
{
    USB0->ENDPOINT[PHY_TO_LOG(endpoint)].ENDPT |= USB_ENDPT_EPSTALL_MASK;
}

void USBHAL::unstallEndpoint(uint8_t endpoint) 
{
    printd("unstall endpoint %d %s\r\n", endpoint>>1,endpoint&1?"TX":"RX");
    ENTER_CRITICAL_SECTION
    {
        USB0->ENDPOINT[PHY_TO_LOG(endpoint)].ENDPT &= ~USB_ENDPT_EPSTALL_MASK;
        int idx = PEP_BDT_IDX(endpoint, 0);
        bdt[idx].info &= ~(BD_OWN_MASK | BD_STALL_MASK | BD_DATA01_MASK);
        
        if (OUT_EP(endpoint))
            Data1 &= ~(1 << endpoint);
        else
            Data1 |= (1 << endpoint);
    }
    EXIT_CRITICAL_SECTION
}

bool USBHAL::getEndpointStallState(uint8_t endpoint) 
{
    uint8_t stall = (USB0->ENDPOINT[PHY_TO_LOG(endpoint)].ENDPT & USB_ENDPT_EPSTALL_MASK);
    return (stall) ? true : false;
}

void USBHAL::remoteWakeup(void) 
{
    // [TODO]
}

// Handle SETUP packet on EP0 IN or OUT
void USBHAL::EP0_SETUP_callback(void)
{
    // Set DATA1 on Control IN endpoint for next packet (recall that we
    // toggle the bit before a send, so clearing the bit sets DATA1 for
    // the next send).  If there's a data IN stage for the SETUP packet,
    // it must always be DATA1, regardless of the prior state of the IN
    // endpoint.
    Data1 &= ~0x02;
    
    // make sure we own the IN enpdoint now, in preparation for the reply
    bdt[PEP_BDT_IDX(EP0IN, EVEN)].info &= ~BD_OWN_MASK;
    
    // process the SETUP packet through the portable protocol code
    EP0setupCallback();
}

// Control endpoint OUT/SETUP callback.  Called from ISR context only.
bool USBHAL::EP0_OUT_callback(void)
{
    int idx = PEP_BDT_IDX(EP0OUT, EVEN);
    if (TOK_PID(idx) == SETUP_TOKEN) 
    {
        // SETUP packet on EP0
        EP0_SETUP_callback();
    } 
    else 
    {
        // OUT packet on EP0 - process it through the protocol code
        EP0out();
    }
    
    // success
    return true;
}

// Control endpoint IN packet handler.  This is only called from ISR context.
bool USBHAL::EP0_IN_callback(void)
{
    int idx = PEP_BDT_IDX(EP0OUT, EVEN);
    if (TOK_PID(idx) == SETUP_TOKEN) 
    {
        // SETUP packet on EP0
        EP0_SETUP_callback();
    } 
    else 
    {
        // process the IN packet through the portable protocol code
        EP0in();
    }
    
    // If we have a SET ADDRESS command outstanding, put it into effect now.
    // The USB spec requires an address change to be made immediately (within 
    // 2ms) after the reply to the SET ADDRESS SETUP packet.  If the flag is
    // set, it means that the EP0in() call above just sent the response, so
    // now is the time to make the address change in the SIE hardware register.
    if (set_addr == 1) 
    {
        USB0->ADDR = addr & 0x7F;
        set_addr = 0;
    }
    
    // success
    return true;
}


void USBHAL::_usbisr(void) 
{
    inIRQ = true;
    instance->usbisr();
    inIRQ = false;
}


void USBHAL::usbisr(void) 
{
    uint8_t i;
    uint8_t istat = USB0->ISTAT;

    // reset interrupt
    if (istat & USB_ISTAT_USBRST_MASK) 
    {
        // disable all endpt
        for(i = 0 ; i < 16 ; i++)
            USB0->ENDPOINT[i].ENDPT = 0x00;

        // enable control endpoint
        realiseEndpoint(EP0OUT, MAX_PACKET_SIZE_EP0, 0);
        realiseEndpoint(EP0IN, MAX_PACKET_SIZE_EP0, 0);

        Data1 = 0x55555555;
        epComplete = 0;
        USB0->CTL |=  USB_CTL_ODDRST_MASK;

        USB0->ISTAT   =  0xFF;  // clear all interrupt status flags
        USB0->ERRSTAT =  0xFF;  // clear all error flags
        USB0->ERREN   =  0xFF;  // enable error interrupt sources
        USB0->ADDR    =  0x00;  // set default address
        
        // notify upper layers of the bus reset
        busReset();
        
        // we're not suspended
        suspendStateChanged(0);
        
        // return now - do no more processing on a RESET interrupt
        return;
    }

    // resume interrupt
    if (istat & USB_ISTAT_RESUME_MASK) 
    {
        suspendStateChanged(0);
        USB0->ISTAT = USB_ISTAT_RESUME_MASK;
    }

    // SOF interrupt
    if (istat & USB_ISTAT_SOFTOK_MASK) 
    {
        // Read frame number and signal the SOF event to the callback
        SOF(frameNumber());
        USB0->ISTAT = USB_ISTAT_SOFTOK_MASK;
    }

    // stall interrupt
    if (istat & USB_ISTAT_STALL_MASK)
    {
        // if the control endpoint (EP 0) is stalled, unstall it
        if (USB0->ENDPOINT[0].ENDPT & USB_ENDPT_EPSTALL_MASK)
        {
            unstallEndpoint(EP0OUT);
            unstallEndpoint(EP0IN);
        }
        
        // clear the SUSPEND flag to allow token processing to continue
        USB0->CTL &= ~USB_CTL_TXSUSPENDTOKENBUSY_MASK;
        USB0->ISTAT = USB_ISTAT_STALL_MASK;
    }

    // token interrupt
    if (istat & USB_ISTAT_TOKDNE_MASK) 
    {
        uint32_t num  = (USB0->STAT >> 4) & 0x0F;
        uint32_t dir  = (USB0->STAT >> 3) & 0x01;
        int endpoint = (num << 1) | dir;
        // uint32_t ev_odd = (USB0->STAT >> 2) & 0x01;  // we only use EVEN buffers, so this is always 0

        // set the Completed bit for the endpoint to indicate that we've
        // finished this send/receive request
        epComplete |= EP(endpoint);
        
        // Call the endpoint packet callback.  If that returns true, it means
        // that the callback handled the packet.  That consumes the packet, so
        // clear the Completed flag to indicate that we're on to the next
        // transaction on the endpoint.
        if ((instance->*(epCallback[endpoint]))())
            epComplete &= ~EP(endpoint);

        // allow token processing to resume
        USB0->CTL &= ~USB_CTL_TXSUSPENDTOKENBUSY_MASK;
        
        // reset the TOKDNE interrupt status flag
        USB0->ISTAT = USB_ISTAT_TOKDNE_MASK;
    }

    // sleep interrupt
    if (istat & USB_ISTAT_SLEEP_MASK) 
    {
        suspendStateChanged(1);
        USB0->ISTAT = USB_ISTAT_SLEEP_MASK;
    }

    // error interrupt
    if (istat & USB_ISTAT_ERROR_MASK) 
    {
        // reset all error status bits, and clear the SUSPEND flag to allow
        // token processing to continue
        USB0->ERRSTAT = 0xFF;
        USB0->CTL &= ~USB_CTL_TXSUSPENDTOKENBUSY_MASK;
        USB0->ISTAT = USB_ISTAT_ERROR_MASK;
    }
}

#endif