Advanced techniques for porting libraries to ESP-IDF components

Introduction

#

In the previous article, “Porting a library to an ESP-IDF component”, we covered the basics of converting an external library into a reusable ESP-IDF component using the simple tinyexpr library as an example. We learned how to create a component structure, handle basic compatibility issues, and suppress compiler warnings at the component level.

However, when porting larger, more complex libraries—especially those originally designed for Linux or other POSIX-compliant systems—you’ll encounter additional challenges that require more advanced techniques. This article builds on the fundamentals from the previous article and explores advanced porting strategies used in real-world library ports such as:

• asio - Asynchronous I/O library
• mosquitto - MQTT broker and client library
• libssh - SSH library
• libwebsockets - WebSockets library

When porting a library, it’s helpful to follow a structured approach:

1. Assess what to reuse and what to change - Identify which parts of the original library can be used as-is and which need adaptation
2. Compose the build system - Create or adapt the CMakeLists.txt to reference the original library sources while integrating port-specific functionality
3. Create the port folder - Set up a dedicated port/ directory structure for adapted functionality that bridges the library and ESP-IDF
4. Adjust functionality - Use advanced techniques like header injection, linker wrapping, and the sock_utils component to adapt the port layer
5. Fine-tune - Manage compiler warnings at the file level and make final adjustments for optimal integration

Now, we will go through these techniques in detail and look at practical examples from real-world ports to help you successfully port complex libraries to ESP-IDF.

Step 1: Assess what to reuse and what to change

#

Before writing any port code, decide what can remain unmodified and what needs adaptation for ESP-IDF. Often, you can keep most of the upstream sources intact and adapt behavior via configuration and thin port layers.

Compile Definitions and Configuration

#

Libraries often use compile-time definitions to enable or disable features, configure behavior, or adapt to different platforms.

Feature Flags and config.h

#

Many libraries use a config.h file generated by autotools or CMake that contains feature detection results. When porting, you’ll need to:

1. Identify which defines are needed
2. Convert them to target_compile_definitions()
3. Make them conditional based on ESP-IDF configuration when appropriate

The asio port demonstrates this pattern:

target_compile_definitions(${COMPONENT_LIB} PUBLIC
    SA_RESTART=0x01
    SA_NOCLDSTOP=0x2
    SA_NOCLDWAIT=0x4
    ASIO_DISABLE_SERIAL_PORT
    ASIO_SEPARATE_COMPILATION
    ASIO_STANDALONE
    ASIO_HAS_PTHREADS
    OPENSSL_NO_ENGINE
    ASIO_DETAIL_IMPL_POSIX_EVENT_IPP)

Some definitions, which are useful for our component to be configurable by Kconfg system could be defined in CMake as conditionals:

if(NOT CONFIG_COMPILER_CXX_EXCEPTIONS)
    target_compile_definitions(${COMPONENT_LIB} PUBLIC ASIO_NO_EXCEPTIONS)
endif()

if(NOT CONFIG_COMPILER_RTTI)
    target_compile_definitions(${COMPONENT_LIB} PUBLIC ASIO_NO_TYPEID)
endif()

This ensures the library adapts to ESP-IDF’s configuration, disabling features that aren’t available (like C++ exceptions or Run-time Type Information – RTTI) when they’re disabled in the project.

Handling Platform Differences

#

Libraries often use #ifdef HAVE_FEATURE patterns to conditionally compile code based on detected features. When porting:

1. Identify which features are available on ESP-IDF
2. Define the appropriate HAVE_* macros
3. Provide implementations or stubs for missing features

For example these macros are used in the mosquitto port:

target_compile_definitions(${COMPONENT_LIB} PRIVATE
    HAVE_PTHREADS=1
    HAVE_SOCKETPAIR=1
    HAVE_PIPE=1)

Step 2: Compose the build system

#

Different libraries use different build systems, and you’ll need to adapt them to ESP-IDF’s CMake-based build system.

Creating Custom CMakeLists.txt

#

Most of the time, you’ll create your own CMakeLists.txt that integrates the library into ESP-IDF’s build system. However, there are different approaches depending on the library:

From Scratch

#

For libraries without CMake support, or when the original CMake is too complex, create a custom CMakeLists.txt that:

• Lists all source files explicitly
• Sets up include directories
• Configures compile definitions
• Manages dependencies

The mosquitto port is an example of this approach.

Include Original CMake

#

Some libraries already have well-structured CMake that can be included directly. The fmt library is an example where the original CMake can be included with minimal modifications:

add_subdirectory(fmt)

Include and Modify Original CMake

#

For libraries with CMake that needs modifications, you can include the original but override specific settings. The mbedtls component uses this approach, including the original CMake but customizing it for ESP-IDF.

Handling Different Build Systems

#

GNU Make

#

Libraries using GNU Make typically require:

• Converting Makefile variables to CMake variables
• Translating build rules to CMake add_library() or idf_component_register()
• Handling conditional compilation manually

Autotools

#

Autotools-based libraries (using configure scripts) are more complex:

• You’ll need to replicate the configuration logic in CMake
• Convert config.h defines to CMake target_compile_definitions()
• Handle feature detection manually

CMake

#

Libraries already using CMake are the easiest:

• Include the original CMake if possible
• Or extract the source lists and recreate the build logic

Source File Management

#

You may need to conditionally include or exclude source files based on ESP-IDF configuration:

set(m_srcs
    ${m_lib_dir}/memory_mosq.c
    ${m_lib_dir}/util_mosq.c
    # ... more sources
)

if(CONFIG_MOSQ_ENABLE_SYS)
    list(APPEND m_srcs ${m_src_dir}/sys_tree.c)
endif()

idf_component_register(SRCS ${m_srcs} ...)

You can also replace sources for testing, as shown in the pre-include example earlier, where test versions of source files replace the originals during unit testing.

Step 3: Create the port folder

#

Create a dedicated port/ directory for adapted functionality that bridges the library with ESP-IDF.

Port-Specific Header Directories

A common pattern in ported libraries is to organize port-specific headers in dedicated directories:

• port/include/ - Public headers that extend or wrap library functionality
• port/priv_include/ - Private headers used only within the port implementation

Both asio and mosquitto use this structure. For example, in the asio port:

idf_component_register(SRCS ${asio_sources}
                    INCLUDE_DIRS "port/include" "asio/asio/include"
                    PRIV_INCLUDE_DIRS ${asio_priv_includes}
                    PRIV_REQUIRES ${asio_requires})

This allows the port to provide custom headers that override or extend the original library’s headers without modifying the upstream source.

Step 4: Adjust functionality (port layer)

#

Header Injection and Pre-inclusion Techniques

#

When porting libraries, you often need to inject custom headers or modify include behavior without changing the original source code. This section covers several techniques for achieving this.

include_next Directive

The include_next directive is a GCC/Clang extension (not C++ specific—it works with C too) that allows you to include the next header in the search path. This is useful for:

• Wrapping or extending standard headers
• Providing compatibility layers
• Adding platform-specific extensions

For example, if you need to extend a standard header:

// port/include/sys/socket.h
#include_next <sys/socket.h>

// Add ESP-IDF specific extensions
int esp_socketpair(int domain, int type, int protocol, int sv[2]);

The include_next directive will find the original sys/socket.h in the system include path and include it, then your custom code adds additional definitions.

Injecting Headers in the Same Directory

Some libraries use old-style include guards that allow you to inject headers in the same directory. This technique works when the library uses patterns like:

#ifndef __timeval_h__
#define __timeval_h__
...
#endif

By placing a header file with the same name in the same directory (or earlier in the include path), you can intercept the include and provide custom definitions. However, this technique is fragile and only works with old-style guards—modern #pragma once headers cannot be intercepted this way.

Pre-include Method

The pre-include method uses the -include compiler flag to automatically include a header before every source file is compiled. This is useful for:

• Injecting dependency definitions
• Providing platform-specific macros
• Replacing standard library functions

Here’s an example pattern from a unit test implementation that uses pre-inclusion to inject test dependencies:

if(CONFIG_MB_UTEST)
    set(dep_inj_dir "${CMAKE_CURRENT_LIST_DIR}/freemodbus/unit_test")
    list(APPEND priv_include_dirs "${dep_inj_dir}/include")
    foreach(src ${srcs})
        get_filename_component(src_wo_ext ${src} NAME_WE)
        if(EXISTS "${dep_inj_dir}/src/${src_wo_ext}_test.c")
            list(REMOVE_ITEM srcs ${src})
            list(APPEND srcs "${dep_inj_dir}/src/${src_wo_ext}_test.c")
            set_property(SOURCE ${src} APPEND_STRING PROPERTY COMPILE_FLAGS
                " -include ${dep_inj_dir}/include/${src_wo_ext}_test.h -include ${dep_inj_dir}/include/dep_inject.h")
        endif()
    endforeach()
endif()

This pattern:

1. Checks if a test version of a source file exists
2. Replaces the original source with the test version
3. Pre-includes test headers that provide mock implementations

Linker Wrapping

#

Linker wrapping is a powerful technique that allows you to override functions without modifying the source code. It uses the --wrap linker flag to redirect function calls to wrapper implementations.

How It Works

#

When you wrap a function function_name, the linker will:

• Redirect calls to function_name() to __wrap_function_name()
• Make the original function available as __real_function_name()

This allows you to:

• Intercept function calls
• Add ESP-IDF specific behavior
• Call the original function when needed
• Completely replace the function if desired

When to use linker wrapping:

• Overriding library functions that need ESP-IDF specific behavior
• Intercepting calls to standard library functions (malloc, free, time, etc.)
• When you want to avoid modifying the original source code
• As an alternative to source-level function replacement

Limitations:

• Only works with functions (not macros or inline functions)
• Requires implementing the wrapper function
• The wrapped function must be linked (not inlined)

Real-World Example: libwebsockets

#

The libwebsockets (lws) port uses linker wrapping to override functions that need ESP-IDF specific behavior. Here’s the CMakeLists.txt pattern:

set(WRAP_FUNCTIONS mbedtls_ssl_handshake_step
                   lws_adopt_descriptor_vhost)

foreach(wrap ${WRAP_FUNCTIONS})
    target_link_libraries(${COMPONENT_LIB} INTERFACE "-Wl,--wrap=${wrap}")
endforeach()

The wrapped functions are then implemented in port/lws_port.c:

extern int __real_mbedtls_ssl_handshake_step(mbedtls_ssl_context *ssl);

int __wrap_mbedtls_ssl_handshake_step(mbedtls_ssl_context *ssl)
{
    int ret = 0;

    while (ssl->MBEDTLS_PRIVATE(state) != MBEDTLS_SSL_HANDSHAKE_OVER) {
        ret = __real_mbedtls_ssl_handshake_step(ssl);

        if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) {
            continue;
        }

        if (ret != 0) {
            break;
        }
    }

    return ret;
}

In this example, the wrapper:

1. Calls the original function via __real_mbedtls_ssl_handshake_step()
2. Handles WANT_READ/WANT_WRITE errors by retrying
3. Continues until the handshake is complete

Socket Utilities (sock_utils) - Deep Dive

#

When porting Linux/Unix libraries to ESP-IDF, you’ll often encounter POSIX socket APIs that aren’t directly available. The sock_utils component provides a compatibility layer that implements common POSIX socket functions using ESP-IDF’s lwIP and esp_netif components.

sock_utils is especially useful when porting libraries that rely on:

• POSIX socket APIs (socketpair, pipe)
• Network interface enumeration (ifaddrs)
• Address resolution (getnameinfo, gai_strerror)
• Hostname resolution (gethostname)

Notes:

• *) socketpair() and pipe() are built on top of lwIP TCP sockets, inheriting the same characteristics. For instance, the maximum transmit buffer size is based on the TCP_SND_BUF setting.
• **) socketpair() and gai_strerror() are declared in sock_utils header files. From ESP-IDF v5.5 onwards, these declarations are automatically propagated to the official header files. If you’re using an older IDF version, you need to manually pre-include the related header files from the sock_utils public include directory.
• ***) pipe() is declared in the compiler’s sys/unistd.h.

Integration Methods

#

For ESP-IDF versions before 5.5, you need to manually ensure the sock_utils headers are included. This can be done by:

1. Adding sock_utils to your component’s REQUIRES or PRIV_REQUIRES:

idf_component_register(SRCS ${sources}
                    PRIV_REQUIRES sock_utils)

2. Pre-including the headers if needed:

target_compile_options(${COMPONENT_LIB} PRIVATE
    "-include" "socketpair.h")

From ESP-IDF v5.5 onwards, the sock_utils declarations are automatically available in the standard headers when you include sock_utils as a dependency. Simply add it to your component:

idf_component_register(SRCS ${sources}
                    PRIV_REQUIRES sock_utils)

Real-World Example: ASIO Using socketpair()

#

The asio library uses socketpair() from sock_utils for its pipe-interrupter implementation. The pipe-interrupter is used to wake up the event loop when needed. Here’s how it’s integrated:

1. The asio component includes sock_utils as a dependency:

set(asio_requires lwip sock_utils)
idf_component_register(SRCS ${asio_sources}
                    PRIV_REQUIRES ${asio_requires})

2. The asio code can then use socketpair() directly, and it will resolve to the sock_utils implementation.

Dependencies

sock_utils requires:

• lwIP - For socket functionality
• esp_netif - For network interface management

These are automatically pulled in when you add sock_utils as a dependency.

Override Functions

#

When porting libraries, you’ll often need to provide implementations for functions that aren’t available on ESP-IDF, or override functions to provide ESP-IDF-specific behavior.

Libraries often need to override standard functions:

• malloc/free - To use ESP-IDF’s memory management
• Time functions - To use ESP-IDF’s time APIs
• Random number generation - To use ESP-IDF’s RNG

These can be implemented as:

• Source-level replacements (providing the function in your port code)
• Linker-wrapped functions (as discussed earlier)
• Pre-included headers that define macros

Stub Function Implementations

#

Some functions may not be needed on ESP-IDF but are required by the library’s API. In these cases, you can provide stub implementations.

The asio port provides a stub for the pause() system call:

extern "C" int pause(void)
{
    while (true) {
        ::sleep(UINT_MAX);
    }
}

This stub is placed in port/src/asio_stub.cpp and provides a minimal implementation that satisfies the library’s requirements.

Step 5: Fine-tune (Compiler Warnings)

#

Component-Level Suppression

#

As covered in the basic porting article, you can suppress compiler warnings for an entire component using target_compile_options():

target_compile_options(${COMPONENT_LIB} PRIVATE -Wno-char-subscripts)

This approach works well when the warning applies broadly across the component, but it can be too broad when only specific files trigger warnings.

File-Level Suppression

#

For more granular control, ESP-IDF allows you to suppress warnings on a per-file basis using set_source_files_properties(). This is particularly useful when:

• Only a few files in a large library trigger warnings
• You want to keep strict warnings enabled for the rest of the component
• Different files need different warning suppressions

The mosquitto library provides a good example of this technique. Some mosquitto source files unconditionally define _GNU_SOURCE, which collides with the ESP-IDF build system and produces redefinition warnings. Instead of suppressing this warning for the entire component, we can target only the offending files:

# Some mosquitto source unconditionally define `_GNU_SOURCE` which collides with IDF build system
# producing warning: "_GNU_SOURCE" redefined
# This workarounds this issue by undefining the macro for the selected files
set(sources_that_define_gnu_source ${m_src_dir}/loop.c ${m_src_dir}/mux_poll.c)
foreach(offending_src ${sources_that_define_gnu_source})
    set_source_files_properties(${offending_src} PROPERTIES COMPILE_OPTIONS "-U_GNU_SOURCE")
endforeach()

You can also use this technique to suppress specific warnings for individual files:

# Suppress format warnings for specific files
set_source_files_properties(${m_src_dir}/logging.c PROPERTIES COMPILE_OPTIONS "-Wno-format")

Or combine multiple options:

set_source_files_properties(${m_src_dir}/file.c PROPERTIES
    COMPILE_OPTIONS "-Wno-format;-Wno-unused-variable")

License Considerations

#

When porting external libraries, always consider license compatibility:

• Check the library’s license - Ensure it’s compatible with your use case
• Maintain attribution - Include license files and copyright notices
• Document modifications - If you modify the library, document the changes
• Consider license of dependencies - Ensure all dependencies are compatible

Most ported libraries maintain the original license and add Espressif’s copyright notice for port-specific code.

Real-World Case Study: ASIO Port

#

The ASIO (Asynchronous I/O) library port demonstrates several advanced techniques working together. Let’s examine the key changes and techniques used.

The ASIO port switched to using the upstream ASIO library directly, reducing maintenance burden while providing a robust asynchronous I/O implementation for ESP-IDF.

Key techniques used:

1. Switching to Upstream

Instead of maintaining a fork, the port now uses the upstream ASIO library directly, making it easier to stay current with updates and bug fixes.

2. Using socketpair() from sock_utils

ASIO’s pipe-interrupter implementation uses socketpair() to create a communication channel for waking the event loop. The port uses sock_utils to provide this POSIX function:

set(asio_requires lwip sock_utils)

3. Pipe-Interrupter Implementation

The pipe-interrupter is a critical component for ASIO’s event loop. By using socketpair() from sock_utils, the port provides a working implementation without modifying ASIO’s source code.

4. Local pause() Stub

ASIO requires the pause() system call, which isn’t available on ESP-IDF. The port provides a stub implementation in port/src/asio_stub.cpp:

extern "C" int pause(void)
{
    while (true) {
        ::sleep(UINT_MAX);
    }
}

5. POSIX Event Customization

The port replaces ASIO’s default posix_event constructor with an ESP-IDF-compatible version that avoids pthread_condattr_t operations that aren’t available on all IDF versions:

// This replaces asio's posix_event constructor
// since the default POSIX version uses pthread_condattr_t operations (init, setclock, destroy),
// which are not available on all IDF versions
posix_event::posix_event()
    : state_(0)
{
    int error = ::pthread_cond_init(&cond_, nullptr);
    asio::error_code ec(error, asio::error::get_system_category());
    asio::detail::throw_error(ec, "event");
}

This is implemented in port/src/asio_stub.cpp and enabled via the ASIO_DETAIL_IMPL_POSIX_EVENT_IPP compile definition.

For more details on the ASIO port changes, see ESP-Protocols PR #717.

Conclusion

#

Porting complex libraries to ESP-IDF requires a combination of techniques:

• Compiler warning management - Use file-level suppression for granular control
• Header injection - Pre-include, include_next, and port directories for seamless integration
• Linker wrapping - Override functions without modifying source code
• sock_utils - Leverage POSIX compatibility for socket-based libraries
• Build system adaptation - Create custom CMake or adapt existing build systems
• Function overrides and stubs - Provide ESP-IDF-specific implementations
• Compile definitions - Configure libraries to work with ESP-IDF’s feature set

The key is to minimize modifications to the original library source code, making it easier to:

• Stay synchronized with upstream updates
• Maintain the port over time
• Share the port with the community

By using these advanced techniques, you can successfully port even complex libraries like ASIO, mosquitto, and libwebsockets to ESP-IDF while maintaining clean, maintainable code.

For more examples, explore the implementations in the ESP-Protocols repository, which contains many ported libraries demonstrating these techniques in practice.