Skip to content

feat(fs): introduce file system abstractions#26

Open
zjw1111 wants to merge 3 commits into
apache:mainfrom
zjw1111:migrate/file-system
Open

feat(fs): introduce file system abstractions#26
zjw1111 wants to merge 3 commits into
apache:mainfrom
zjw1111:migrate/file-system

Conversation

@zjw1111
Copy link
Copy Markdown
Contributor

@zjw1111 zjw1111 commented May 28, 2026

Purpose

Linked issue: None

Migrate the basic file system abstractions from the Alibaba-origin C++ repository into Apache Paimon C++.

Requested files migrated:

  • include/paimon/fs/file_system.h
  • include/paimon/fs/file_system_factory.h
  • src/paimon/common/fs/file_system.cpp
  • src/paimon/common/fs/file_system_factory.cpp
  • src/paimon/common/fs/resolving_file_system.h
  • src/paimon/common/fs/resolving_file_system.cpp

Extra dependency files migrated for a complete build closure:

  • include/paimon/factories/factory.h
  • include/paimon/factories/factory_creator.h
  • src/paimon/common/factories/factory_creator.cpp
  • include/paimon/factories/singleton.h
  • src/paimon/common/factories/singleton.cpp
  • src/paimon/common/factories/io_hook.h
  • src/paimon/common/factories/io_hook.cpp

License handling:

  • Replaced Alibaba-owned file headers with ASF headers.
  • Preserved the Alibaba Havenask-derived headers in singleton.h and singleton.cpp.
  • Added the corresponding Havenask declaration to top-level LICENSE and NOTICE.

External contributor handling:

  • Final blame analysis found no contributor above the automatic or file-level threshold.
  • No Co-authored-by trailer was added.

Tests

  • git diff --check
  • clang-format on migrated C++ files
  • Migration preflight: no missing internal dependencies
  • External contributor analysis on the final migrated file list

Note: cmake --build build -j64 could not run in this checkout because the existing build/ directory has no generated Makefile or other usable build generator files.

API and Format

This adds public file system and factory headers under include/paimon/. It does not change storage format or protocol.

Documentation

No user-facing documentation changes.

Generative AI tooling

Migrate-by: OpenAI Codex

Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for migrating the file system abstractions. I found a blocking build/link issue in the current head.

Several newly added utility APIs are only declared, but no implementation is added in this PR. For example:

  • Path::ToString, PathUtil::CreateTempPath, and PathUtil::ToPath are declared in src/paimon/common/utils/path_util.h, and are called by the new file system code.
  • StringUtils::StartsWith, StringUtils::EndsWith, and StringUtils::ToLowerCase are declared in src/paimon/common/utils/string_utils.h, and existing/new code calls them from data_type_json_parser.cpp, blob_utils.cpp, and file_system.cpp.

I could not find any corresponding definitions in the PR. This means the code either fails to link as soon as those call sites are included in a target, or fails to build if these utilities are expected to be header-only. Please add the missing definitions (or inline the functions in the headers) and include tests/at least a build target that exercises the new file system and utility code.

I also noticed that this PR does not add any tests for the new path parsing / resolving file system behavior, so the cache key, default scheme mapping, and helper functions are currently unverified.

@zjw1111 zjw1111 changed the title feat(fs): migrate file system abstractions feat(fs): introduce file system abstractions May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants