add Win32_CODEPAGES enum and enable utf8 console output on win32

[Book-reader]

this adds an enum of all the win32 codepages from https://learn.microsoft.com/en-us/windows/win32/intl/code-page-identifiers and sets the output codepage to utf8 with an @init function so that utf8 characters can be printed correctly (see #2668)

[Sorrow-Scarlet]

I also found this when compile sth with comments:

15:     //鍒濆鍖?
16: 
17:     //杩愯鏈嶅姟鍣?
18:     while(true)
        ^^^^^

Reproduce this by return sth after a while loop when you do utf-8 comments

[Book-reader]

What does the full error message say?

[lerno]

Instead of making that an @init maybe it should be baked into main?

[Sorrow-Scarlet]

I just mean you can find garbles here, not an error, but a warning. I returned a pointer after a While true. 发自我的 iPhone 在 2025年12月23日，11:23，Book-reader ***@***.***> 写道：  [https://avatars.githubusercontent.com/u/89329785?s=20&v=4]Book-reader left a comment (c3lang/c3c#2670)<#2670 (comment)> What does the full error message say? — Reply to this email directly, view it on GitHub<#2670 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOWKQ2SF4ENPE2K6CT22SNL4DCYSJAVCNFSM6AAAAACPWZSCP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOBUHEZTMMBVGY>. You are receiving this because you commented.Message ID: ***@***.***>

[Book-reader]

Instead of making that an @init maybe it should be baked into main?

I've added it to all the main function wrappers, I'm not sure if that's exactly what you meant though

[Book-reader]

I just mean you can find garbles here, not an error, but a warning.

Oh I see, you mean the text in the comments are garbled in the error message. I can add something to try to fix that as well.

[Book-reader]

@Sorrow-Scarlet when c3-windows-Release is added to the Artifacts section at the bottom of the https://github.com/c3lang/c3c/actions/runs/20461387289 page (it might take a few hours), could you try downloading it and see if it still garbles the comments in the error messages?

[Sorrow-Scarlet]

Sure, I'll try it now 发自我的 iPhone 在 2025年12月23日，21:12，Book-reader ***@***.***> 写道：  [https://avatars.githubusercontent.com/u/89329785?s=20&v=4]Book-reader left a comment (c3lang/c3c#2670)<#2670 (comment)> @Sorrow-Scarlet<https://github.com/Sorrow-Scarlet> when c3-windows-Release is added to the Artifacts section at the bottom of the https://github.com/c3lang/c3c/actions/runs/20461387289 page (it might take a few hours), could you try downloading it and see if it still garbles the comments in the error messages? — Reply to this email directly, view it on GitHub<#2670 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOWKQ2VE4ECMVI5TQ5BKRDT4DE5S3AVCNFSM6AAAAACPWZSCP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOBWGU4TIMJZG4>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Sorrow-Scarlet]

@Sorrow-Scarlet when c3-windows-Release is added to the Artifacts section at the bottom of the https://github.com/c3lang/c3c/actions/runs/20461387289 page (it might take a few hours), could you try downloading it and see if it still garbles the comments in the error messages?

I found CI artifact download at extreme slow spd:44kb/s, do you know the reason? (Normal CI and release download fastly as 10mb/s)

[Book-reader]

I found CI artifact download at extreme slow spd:44kb/s, do you know the reason? (Normal CI and release download fastly as 10mb/s)

I think that's github saving money by using a slower connection or not distributing it globally with a CDN because CI artifacts are mostly for testing and very few people will download them, so they would be wasting space in the CDN that could be used for more commonly downloaded things like releases.

[Sorrow-Scarlet]

I think that's github saving money by using a slower connection or not distributing it globally with a CDN because CI artifacts are mostly for testing and very few people will download them, so they would be wasting space in the CDN that could be used for more commonly downloaded things like releases.

Well...I see. I found this when running c3c build

$ c3c build
1: module test;
2:
3: fn void main(String[] args)
           ^^^^
(C:/Users/Sorrow/Downloads/tempproject/test/src/main.c3:3:9) Error: Missing main forwarding function '@wmain_to_void_main_args'. 

Code:

module test;

fn void main(String[] args)
{
    int a = 0 ;
    while(1)
    {
        //测试是否乱码
        //test whether garble
        a++;
    }
	int b;
}

[Book-reader]

Right, I'd forgotten that the CI for windows doesn't include the standard library, which is needed for most main functions. You'll also need to to download https://github.com/Book-reader/c3c/tree/win32_codepages as a zip and copy the lib folder into the same folder as c3c.exe

[Sorrow-Scarlet]

Right, I'd forgotten that the CI for windows doesn't include the standard library, which is needed for most main functions. You'll also need to to download https://github.com/Book-reader/c3c/tree/win32_codepages as a zip and copy the lib folder into the same folder as c3c.exe

OK, I'm downloading

[Sorrow-Scarlet]

@Book-reader Awesome!

$c3cbuild                                                                                                                          
 8:         //测试是否乱码                                                                                                           
 9:         a++;                                                                                                                     
10:     }
11:     int b;
        ^^^^^
(C:/Users/Sorrow/Downloads/tempproject/test/src/main.c3:11:2) Warning: This code will never execute.

 3: fn void main(String[] args)
 4: {
 5:     int a = 0 ;
 6:     while(1)
        ^^^^^
(C:/Users/Sorrow/Downloads/tempproject/test/src/main.c3:6:5) Note: No code will execute after this statement.

[Book-reader]

Great! Does printing it with io::printfn also work now without needing to run chcp or setConsoleOutputCP?

[Book-reader]

And another thing, if you read a line of chinese text from the console with io::readline(mem) and then print it, does it get printed correctly?

[Sorrow-Scarlet]

And another thing, if you read a line of chinese text from the console with io::readline(mem) and then print it, does it get printed correctly?

I'm checking this and the previous now

[Sorrow-Scarlet]

@Book-reader Oh no!

Test with 5 characters

Code:

module test;
import std::io;

fn int main(String[] args)
{
    //console output
	io::printn("控制台测试");
	return 0;
}

Output:

$c3c build
$.\build\test.exe
鎺у埗鍙版祴璇

Test with 2 characters

Code:

module test;
import std::io;

fn int main(String[] args)
{
    //console output
	io::printn("测试");
	return 0;
}

Output:

$c3c build
$.\build\test.exe
娴嬭瘯

[Sorrow-Scarlet]

@Book-reader

Test io::printfn()

Bad!

Code:

module test;
import std::io;

fn void main(String[] args)
{
    String s = "控制台测试";
    io::printfn("%s",s);

}

Output:

$ c3c build
$ .\build\test.exe
鎺у埗鍙版祴璇?

Test io::readline(mem)

Perfect!

Code:

module test;
import std::io;

fn void main(String[] args)
{
    String s = io::readline(mem)!!;
    io::print(s);
}

Output:

$ c3c build
$ .\build\test.exe
控制台测试 //input
控制台测试 //output

Test io::treadline()

Perfect!

Code:

module test;
import std::io;

fn void main(String[] args)
{
    String s = io::treadline()!!;
    io::print(s);
}

Output:

$c3c build
$.\build\test.exe
测试 //input
测试 //output

[Book-reader]

@Sorrow-Scarlet and this is on the version you downloaded from the github actions earlier right? Something must not have worked then.
Just to make sure, can you upload the contents of the lib/std/core/private/main_stub.c3 file relative to the c3c.exe you ran?

[Sorrow-Scarlet]

@Sorrow-Scarlet and this is on the version you downloaded from the github actions earlier right? Something must not have worked then. Just to make sure, can you upload the contents of the lib/std/core/private/main_stub.c3 file relative to the c3c.exe you ran?

Github don't support C3 file directly, so I changed it into txt
File:
main_stub.txt

One more bad thing, after using https://github.com/Book-reader/c3c/tree/win32_codepages lib, this code output garbles. (0.7.8release lib is fine)

Code:

module test;
import std::io;

fn void main()
{
    io::print("Enter a text:");
    String text=io::treadline().trim()!!;
    io::printfn("You entered %s",text);
}

// lib from release
$ .\build\test.exe
Enter a text:测试
You entered 测试

// lib from https://github.com/Book-reader/c3c/tree/win32_codepages
$ c3c build
$ .\build\test.exe
Enter a text:测试
You entered ����

[Sorrow-Scarlet]

@Book-reader Oh... I might mistake sth...

The previous bad tests, all print function returns right with your lib fork and CI artifact.

But print received strings returned garbles, this work find with release lib and CI artifact.

[Book-reader]

Ok, that's good to know. The reason printing recieved strings isn't working is because it is printing the text in a different codepage to how it reads it, so I'll upload a new version that sets the input codepage to utf8 as well which should fix it.

[Sorrow-Scarlet]

Ok, that's good to know. The reason printing recieved strings isn't working is because it is printing the text in a different codepage to how it reads it, so I'll upload a new version that sets the input codepage to utf8 as well which should fix it.

Two compilers and two libs really drives me crazy, I even forgot which to use. Hope there would be a C3 Manager in the future.

[Book-reader]

Two compilers and two libs really drives me crazy, I even forgot which to use. Hope there would be a C3 Manager in the future.

This isn't usually as big of a problem on other operating systems because the CI usually includes the standard library in the download as well, I don't know why it isn't included on windows (@lerno do you know why?).

I've updated the standard library at https://github.com/Book-reader/c3c/tree/win32_codepages to fix printing recieved strings, could you download it again and see if it works now? (you can delete the old version you downloaded from there)
Thank you for helping me test this by the way, I don't have a windows computer so I wouldn't have been able to find all the problems myself.

[Sorrow-Scarlet]

This isn't usually as big of a problem on other operating systems because the CI usually includes the standard library in the download as well, I don't know why it isn't included on windows (@lerno do you know why?).

I've updated the standard library at https://github.com/Book-reader/c3c/tree/win32_codepages to fix printing recieved strings, could you download it again and see if it works now? (you can delete the old version you downloaded from there) Thank you for helping me test this by the way, I don't have a windows computer so I wouldn't have been able to find all the problems myself.

YES, fixed now.

$ c3c build
$ .\build\test.exe                                                                                                  
Enter a text:测试                                                                                                   
You entered 测试  

[lerno]

This isn't usually as big of a problem on other operating systems because the CI usually includes the standard library in the download as well, I don't know why it isn't included on windows (@lerno do you know why?).

It is included. I am not sure why you're saying it's not.

[lerno]

Note that there are two ways windows have a forwarding function. But when it sees fn int main() it will not create any forwarding. This means that in this case there will be no call to the function updating the codepage. So I think this needs to be addressed as well. I feel that the macro for creating fn void main() is a bit annoying that it's missing when you compile without the stdlib. However, creating arguments might understandably need to be extra code.

If you look at sema_decl in sema_create_synthetic_main, you can see all the different macros used, which match the macros in main_stub.c3. One could consider whether more could be lifted into the sema, instead relying on the helper methods. Alternatively, try to use the macros for all cases, only backing off in the case there is no macro and it's possible to implement it without inlining the main function, e.g. in the fn int main() case. I am not sure.

[Book-reader]

It is included. I am not sure why you're saying it's not.

It's added when creating a pre-release tag, but not during regular CI, so downloading a windows CI artifact from a PR to test it won't include the standard library
https://github.com/c3lang/c3c/blob/master/.github/workflows/main.yml#L107-L113
https://github.com/c3lang/c3c/blob/master/.github/workflows/main.yml#L1183-L1184

[Book-reader]

Note that there are two ways windows have a forwarding function. But when it sees fn int main() it will not create any forwarding. This means that in this case there will be no call to the function updating the codepage. So I think this needs to be addressed as well. I feel that the macro for creating fn void main() is a bit annoying that it's missing when you compile without the stdlib. However, creating arguments might understandably need to be extra code.

If you look at sema_decl in sema_create_synthetic_main, you can see all the different macros used, which match the macros in main_stub.c3. One could consider whether more could be lifted into the sema, instead relying on the helper methods. Alternatively, try to use the macros for all cases, only backing off in the case there is no macro and it's possible to implement it without inlining the main function, e.g. in the fn int main() case. I am not sure.

I'll have a look at that

[lerno]

It is included. I am not sure why you're saying it's not.

It's added when creating a pre-release tag, but not during regular CI, so downloading a windows CI artifact from a PR to test it won't include the standard library https://github.com/c3lang/c3c/blob/master/.github/workflows/main.yml#L107-L113 https://github.com/c3lang/c3c/blob/master/.github/workflows/main.yml#L1183-L1184

Well, you don't download the artifact normally, do you? I would assume people downloaded the lastest prerelease? Everything else is considered intermediate files.

[Book-reader]

That's fair, and most people who test a PR would usually clone & build it themselves so it isn't a problem normally