Upgrade embedded Python from 3.12 to 3.14 #135

New Issue

john · 2025-11-26T15:45:10Z

john commented

2025-11-26 15:45:10 +00:00

Overview

Upgrade the embedded Python interpreter from 3.12.2 to 3.14.0 (commit ebf955d). This is a significant upgrade spanning two major versions with breaking changes in both the Python API and C API.

Current State

Current version: Python 3.12.2 (commit 6abddd9)
Target version: Python 3.14.0 (commit ebf955d)
Submodule path: modules/cpython

Breaking Changes Analysis

Python 3.13 C API Changes (must address)

Deprecated APIs used by McRogueFace:

PySys_SetArgvEx() - Deprecated in 3.11
- Location: src/main.cpp:146
- Current usage:
```
PySys_SetArgvEx(config.script_args.size() + 1, python_argv, 0);
```
- Replacement: Use PyConfig API (PEP 587/741)
```
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.argv = ...;  // Set argv via PyConfig
Py_InitializeFromConfig(&config);
```
Py_InspectFlag - Deprecated in 3.12
- Location: src/main.cpp:191
- Current usage:
```
Py_InspectFlag = 1;
```
- Replacement: Use PyConfig.inspect or new PyInitConfig API (PEP 741)
Header changes - Python.h no longer includes:
- <time.h> - provides clock(), gmtime()
- <sys/select.h> - provides select()
- <sys/time.h> - provides futimes(), gettimeofday(), setitimer()
- <stddef.h> (Windows only) - provides offsetof(), size_t, ptrdiff_t
Action: Audit all .cpp files for implicit use of these headers
Removed private APIs (check if used):
- _PyDict_Pop() → PyDict_Pop()
- _PyLong_AsInt() → PyLong_AsInt()
- _PyThreadState_UncheckedGet() → PyThreadState_GetUnchecked()
- _Py_IsFinalizing() → Py_IsFinalizing()
Trashcan macros (if used in custom types):
- Py_TRASHCAN_SAFE_BEGIN/Py_TRASHCAN_SAFE_END removed
- Replace with Py_TRASHCAN_BEGIN/Py_TRASHCAN_END

Python 3.14 C API Changes

New PyInitConfig API (PEP 741):
- PyInitConfig_Create(), PyInitConfig_Free()
- PyInitConfig_SetInt(), PyInitConfig_SetStr()
- Py_InitializeFromInitConfig()
- This is the preferred way to configure Python initialization
Limited C API changes:
- Py_TYPE() and Py_REFCNT() now opaque function calls
- PySequence_Fast_GET_SIZE, PySequence_Fast_GET_ITEM, PySequence_Fast_ITEMS removed
New useful APIs:
- PyBytes_Join() - like sep.join(iterable)
- PyUnicode_Equal() - test string equality
- PyLong_IsPositive(), PyLong_IsNegative(), PyLong_IsZero()
- PyIter_NextItem() - replaces ambiguous PyIter_Next()

Build System Changes

C11 atomics required: Compiler must support C11 atomic library
Autoconf 2.72 required (for regenerating configure)
SQLite 3.15.2+ required for sqlite3 module
mimalloc bundled by default (MIT license)
--with-system-libmpdec now defaults to yes

McRogueFace Code Audit

Files requiring changes:

src/main.cpp:

// Line 146 - Replace PySys_SetArgvEx
PySys_SetArgvEx(config.script_args.size() + 1, python_argv, 0);

// Line 191 - Replace Py_InspectFlag  
Py_InspectFlag = 1;

Files to audit for private API usage:

src/McRFPy_API.cpp
src/PyColor.cpp
src/PyTexture.cpp
src/PyFont.cpp
src/UIFrame.cpp
src/UICaption.cpp
src/UISprite.cpp
src/UIGrid.cpp
src/UIEntity.cpp
All other Py*.cpp files

Grep patterns to check:

# Private APIs
grep -r "_Py[A-Z]" src/

# Deprecated trashcan
grep -r "Py_TRASHCAN_SAFE" src/

# Deprecated flags
grep -r "Py_.*Flag" src/

# Header dependencies
grep -r "time\.h\|sys/select\.h\|sys/time\.h" src/

Implementation Strategy

Option A: Direct upgrade (recommended if changes are minimal)

Update submodule to Python 3.14
Fix deprecated API usage
Add missing header includes
Test and fix any remaining issues

Option B: Incremental upgrade

First upgrade to Python 3.13
Fix 3.13 deprecations
Then upgrade to Python 3.14
Fix 3.14 deprecations

Option C: Use pythoncapi-compat

The pythoncapi-compat project provides compatibility shims
Can help maintain backwards compatibility with older Python versions
May simplify the transition

Tasks

Audit codebase for deprecated API usage (grep patterns above)
Audit codebase for implicit header dependencies
Update src/main.cpp to use PyConfig/PyInitConfig API
Update submodule to Python 3.14.0 (ebf955d)
Fix compilation errors
Run test suite
Update BUILD_FROM_SOURCE.md with any new build requirements
Consider pythoncapi-compat for backward compatibility

Testing Requirements

After upgrade:

# Basic functionality
./mcrogueface --version
./mcrogueface --headless -c "import mcrfpy; print('OK')"

# Script execution with args
./mcrogueface --headless --exec test_script.py arg1 arg2

# Interactive mode
./mcrogueface -i

# Full test suite
cd tests && python3 run_tests.py

References

Python 3.13 What's New: https://docs.python.org/3.13/whatsnew/3.13.html
Python 3.14 What's New: https://docs.python.org/3.14/whatsnew/3.14.html
PEP 587 (PyConfig): https://peps.python.org/pep-0587/
PEP 741 (PyInitConfig): https://peps.python.org/pep-0741/
pythoncapi-compat: https://github.com/python/pythoncapi-compat
Target commit: ebf955d

## Overview Upgrade the embedded Python interpreter from 3.12.2 to 3.14.0 (commit ebf955d). This is a significant upgrade spanning two major versions with breaking changes in both the Python API and C API. ## Current State - **Current version**: Python 3.12.2 (commit 6abddd9) - **Target version**: Python 3.14.0 (commit ebf955d) - **Submodule path**: `modules/cpython` ## Breaking Changes Analysis ### Python 3.13 C API Changes (must address) 1. **Deprecated APIs used by McRogueFace**: **`PySys_SetArgvEx()`** - Deprecated in 3.11 - Location: `src/main.cpp:146` - Current usage: ```cpp PySys_SetArgvEx(config.script_args.size() + 1, python_argv, 0); ``` - Replacement: Use `PyConfig` API (PEP 587/741) ```cpp PyConfig config; PyConfig_InitPythonConfig(&config); config.argv = ...; // Set argv via PyConfig Py_InitializeFromConfig(&config); ``` **`Py_InspectFlag`** - Deprecated in 3.12 - Location: `src/main.cpp:191` - Current usage: ```cpp Py_InspectFlag = 1; ``` - Replacement: Use `PyConfig.inspect` or new PyInitConfig API (PEP 741) 2. **Header changes** - `Python.h` no longer includes: - `<time.h>` - provides `clock()`, `gmtime()` - `<sys/select.h>` - provides `select()` - `<sys/time.h>` - provides `futimes()`, `gettimeofday()`, `setitimer()` - `<stddef.h>` (Windows only) - provides `offsetof()`, `size_t`, `ptrdiff_t` **Action**: Audit all `.cpp` files for implicit use of these headers 3. **Removed private APIs** (check if used): - `_PyDict_Pop()` → `PyDict_Pop()` - `_PyLong_AsInt()` → `PyLong_AsInt()` - `_PyThreadState_UncheckedGet()` → `PyThreadState_GetUnchecked()` - `_Py_IsFinalizing()` → `Py_IsFinalizing()` 4. **Trashcan macros** (if used in custom types): - `Py_TRASHCAN_SAFE_BEGIN`/`Py_TRASHCAN_SAFE_END` removed - Replace with `Py_TRASHCAN_BEGIN`/`Py_TRASHCAN_END` ### Python 3.14 C API Changes 1. **New PyInitConfig API** (PEP 741): - `PyInitConfig_Create()`, `PyInitConfig_Free()` - `PyInitConfig_SetInt()`, `PyInitConfig_SetStr()` - `Py_InitializeFromInitConfig()` - This is the preferred way to configure Python initialization 2. **Limited C API changes**: - `Py_TYPE()` and `Py_REFCNT()` now opaque function calls - `PySequence_Fast_GET_SIZE`, `PySequence_Fast_GET_ITEM`, `PySequence_Fast_ITEMS` removed 3. **New useful APIs**: - `PyBytes_Join()` - like `sep.join(iterable)` - `PyUnicode_Equal()` - test string equality - `PyLong_IsPositive()`, `PyLong_IsNegative()`, `PyLong_IsZero()` - `PyIter_NextItem()` - replaces ambiguous `PyIter_Next()` ### Build System Changes 1. **C11 atomics required**: Compiler must support C11 atomic library 2. **Autoconf 2.72 required** (for regenerating configure) 3. **SQLite 3.15.2+ required** for sqlite3 module 4. **mimalloc bundled by default** (MIT license) 5. **`--with-system-libmpdec` now defaults to yes** ## McRogueFace Code Audit ### Files requiring changes: **`src/main.cpp`**: ```cpp // Line 146 - Replace PySys_SetArgvEx PySys_SetArgvEx(config.script_args.size() + 1, python_argv, 0); // Line 191 - Replace Py_InspectFlag Py_InspectFlag = 1; ``` ### Files to audit for private API usage: - `src/McRFPy_API.cpp` - `src/PyColor.cpp` - `src/PyTexture.cpp` - `src/PyFont.cpp` - `src/UIFrame.cpp` - `src/UICaption.cpp` - `src/UISprite.cpp` - `src/UIGrid.cpp` - `src/UIEntity.cpp` - All other `Py*.cpp` files ### Grep patterns to check: ```bash # Private APIs grep -r "_Py[A-Z]" src/ # Deprecated trashcan grep -r "Py_TRASHCAN_SAFE" src/ # Deprecated flags grep -r "Py_.*Flag" src/ # Header dependencies grep -r "time\.h\|sys/select\.h\|sys/time\.h" src/ ``` ## Implementation Strategy ### Option A: Direct upgrade (recommended if changes are minimal) 1. Update submodule to Python 3.14 2. Fix deprecated API usage 3. Add missing header includes 4. Test and fix any remaining issues ### Option B: Incremental upgrade 1. First upgrade to Python 3.13 2. Fix 3.13 deprecations 3. Then upgrade to Python 3.14 4. Fix 3.14 deprecations ### Option C: Use pythoncapi-compat - The [pythoncapi-compat](https://github.com/python/pythoncapi-compat) project provides compatibility shims - Can help maintain backwards compatibility with older Python versions - May simplify the transition ## Tasks - [ ] Audit codebase for deprecated API usage (grep patterns above) - [ ] Audit codebase for implicit header dependencies - [ ] Update `src/main.cpp` to use PyConfig/PyInitConfig API - [ ] Update submodule to Python 3.14.0 (ebf955d) - [ ] Fix compilation errors - [ ] Run test suite - [ ] Update BUILD_FROM_SOURCE.md with any new build requirements - [ ] Consider pythoncapi-compat for backward compatibility ## Testing Requirements After upgrade: ```bash # Basic functionality ./mcrogueface --version ./mcrogueface --headless -c "import mcrfpy; print('OK')" # Script execution with args ./mcrogueface --headless --exec test_script.py arg1 arg2 # Interactive mode ./mcrogueface -i # Full test suite cd tests && python3 run_tests.py ``` ## References - Python 3.13 What's New: https://docs.python.org/3.13/whatsnew/3.13.html - Python 3.14 What's New: https://docs.python.org/3.14/whatsnew/3.14.html - PEP 587 (PyConfig): https://peps.python.org/pep-0587/ - PEP 741 (PyInitConfig): https://peps.python.org/pep-0741/ - pythoncapi-compat: https://github.com/python/pythoncapi-compat - Target commit: ebf955d

john referenced this issue from a commit

2025-11-26 23:19:42 +00:00

feat: Migrate to Python 3.14 (closes #135)

john closed this issue

2025-11-26 23:19:42 +00:00

Sign in to join this conversation.