Bookstore

 Computer Math and Games in Pascal (preview) Lazarus Handbook

Author Topic: Range checking Enum types. Unreliable or confusing?  (Read 1004 times)

derek.john.evans

• Jr. Member
• Posts: 55
Range checking Enum types. Unreliable or confusing?
« on: July 30, 2022, 05:29:23 am »

I've been looking for a better (safer) way to cast integers to enum types. I thought I understood "range checking" but after some testing, it appears I don't. I've attached the source to the tests I've been trying.

So, lets say you have an enum type:
Code: Pascal  [Select][+][-]
1. TEnum_All = (opt0, opt1, opt2, opt3, opt4);

And you then define a sub range of this type:
Code: Pascal  [Select][+][-]
1. TEnum_1_3 = opt1..opt3;

And, for completeness, lets define a type which includes all of the TEnum_All values:
Code: Pascal  [Select][+][-]
1. TEnum_0_4 = opt0..opt4;

When casting from an integer to one of my enum types, there are 3 possible casting situations, and 3 possible variable destinations. meaning 3x3 = 9 tests.

Code: Pascal  [Select][+][-]
1. program EnumRangeCheckTesting;
2.
3. {\$mode objfpc}{\$H+}{\$R+}
4.
5. uses
6.   SysUtils;
7.
8. type
9.
10.   TEnum_All = (opt0, opt1, opt2, opt3, opt4);
11.   TEnum_1_3 = opt1..opt3;
12.   TEnum_0_4 = opt0..opt4;
13.
14. var
15.   IntValue: integer;
16.   Enum_All: TEnum_All;
17.   Enum_1_3: TEnum_1_3;
18.   Enum_0_4: TEnum_0_4;
19.
20.   procedure Group1_1;
21.   begin
22.     Write('Enum_All := TEnum_All(IntValue)');
23.     Enum_All := TEnum_All(IntValue);
24.   end;
25.
26.   procedure Group1_2;
27.   begin
28.     Write('Enum_1_3 := TEnum_All(IntValue)');
29.     Enum_1_3 := TEnum_All(IntValue);
30.   end;
31.
32.   procedure Group1_3;
33.   begin
34.     Write('Enum_0_4 := TEnum_All(IntValue)');
35.     Enum_0_4 := TEnum_All(IntValue);
36.   end;
37.
38.   procedure Group2_1;
39.   begin
40.     Write('Enum_All := TEnum_1_3(IntValue)');
41.     Enum_All := TEnum_1_3(IntValue);
42.   end;
43.
44.   procedure Group2_2;
45.   begin
46.     Write('Enum_1_3 := TEnum_1_3(IntValue)');
47.     Enum_1_3 := TEnum_1_3(IntValue);
48.   end;
49.
50.   procedure Group2_3;
51.   begin
52.     Write('Enum_0_4 := TEnum_1_3(IntValue)');
53.     Enum_0_4 := TEnum_1_3(IntValue);
54.   end;
55.
56.   procedure Group3_1;
57.   begin
58.     Write('Enum_All := TEnum_0_4(IntValue)');
59.     Enum_All := TEnum_0_4(IntValue);
60.   end;
61.
62.   procedure Group3_2;
63.   begin
64.     Write('Enum_1_3 := TEnum_0_4(IntValue)');
65.     Enum_1_3 := TEnum_0_4(IntValue);
66.   end;
67.
68.   procedure Group3_3;
69.   begin
70.     Write('Enum_0_4 := TEnum_0_4(IntValue)');
71.     Enum_0_4 := TEnum_0_4(IntValue);
72.   end;
73.
74.   procedure TestProc(ATestProc: TProcedure);
75.   begin
76.     try
77.       ATestProc;
78.       WriteLn(' Success');
79.     except
80.       on E: Exception do WriteLn(' Failed: ', E.Message);
81.     end;
82.   end;
83.
84. begin
85.   IntValue := 1234; (* Integer IntValue is outside all enum ranges *)
86.
87.   Writeln('Group 1');
88.   TestProc(@Group1_1);
89.   TestProc(@Group1_2);
90.   TestProc(@Group1_3);
91.   WriteLn;
92.
93.   Writeln('Group 2');
94.   TestProc(@Group2_1);
95.   TestProc(@Group2_2);
96.   TestProc(@Group2_3);
97.   WriteLn;
98.
99.   Writeln('Group 3');
100.   TestProc(@Group3_1);
101.   TestProc(@Group3_2);
102.   TestProc(@Group3_3);
103.   WriteLn;
104.
106. end.

Running this with an integer value of 1234, I get:

Group 1
Enum_All := TEnum_All(IntValue) Success
Enum_1_3 := TEnum_All(IntValue) Failed: Range check error
Enum_0_4 := TEnum_All(IntValue) Failed: Range check error

Group 2
Enum_All := TEnum_1_3(IntValue) Failed: Range check error
Enum_1_3 := TEnum_1_3(IntValue) Success
Enum_0_4 := TEnum_1_3(IntValue) Failed: Range check error

Group 3
Enum_All := TEnum_0_4(IntValue) Failed: Range check error
Enum_1_3 := TEnum_0_4(IntValue) Failed: Range check error
Enum_0_4 := TEnum_0_4(IntValue) Success

Which means, the most common enum type casting doesn't raise a range check error (?)
Interestingly the case Enum_All := TEnum_0_4(IntValue) fails, while Enum_All := TEnum_All(IntValue) succeeds (?)

I'm sure all of this is perfectly logical, but all I can deduce from my tests is, I cant rely of range checking to protect my code from illegal enum castings.

My main question is, how are other people handling enum type casting from external sources (ie: db tables, config files, etc)? I'm currently manually checking the integer is in range.

note: I have come across forum requests for language features to make "int to enum casting" safer and easier.

Sorry. I forgot, Im using Lazarus 2.0.10 (Raspberry Pi 400)
« Last Edit: July 30, 2022, 05:42:27 am by derek.john.evans »

• Hero Member
• Posts: 11773
Re: Range checking Enum types. Unreliable or confusing?
« Reply #1 on: July 30, 2022, 06:49:55 am »
Check my wiki entry:
https://wiki.freepascal.org/Defensive_programming_techniques
It explains to some length how you should handle this.
Black themes should be banned.

derek.john.evans

• Jr. Member
• Posts: 55
Re: Range checking Enum types. Unreliable or confusing?
« Reply #2 on: July 30, 2022, 07:53:13 am »
Check my wiki entry:
https://wiki.freepascal.org/Defensive_programming_techniques
It explains to some length how you should handle this.

Thanks Thaddy for the reply. I use most of the techniques on that page. My post is part of a larger idea. I wanted to verify that range checking was not going to identify bugs before I proposed the following idea:

Code: Pascal  [Select][+][-]
1. generic function safe_enum<T>(AValue: integer): T;
2. begin
3.   if (AValue < Ord(Low(T))) or (AValue > Ord(High(T))) then begin
4.     raise ERangeError.Create('safe_enum range check error');
5.   end;
6.   Result := T(AValue);
7. end;

The updated testing code is now:

Code: Pascal  [Select][+][-]
1. program EnumRangeCheckTesting;
2.
3. {\$mode objfpc}{\$H+}{\$R+}
4.
5. uses
6.   SysUtils;
7.
8. generic function safe_enum<T>(AValue: integer): T;
9. begin
10.   if (AValue < Ord(Low(T))) or (AValue > Ord(High(T))) then begin
11.     raise ERangeError.Create('safe_enum range check error');
12.   end;
13.   Result := T(AValue);
14. end;
15.
16. type
17.
18.   TEnum_All = (opt0, opt1, opt2, opt3, opt4);
19.   TEnum_1_3 = opt1..opt3;
20.   TEnum_0_4 = opt0..opt4;
21.
22. var
23.   IntValue: integer;
24.   Enum_All: TEnum_All;
25.   Enum_1_3: TEnum_1_3;
26.   Enum_0_4: TEnum_0_4;
27.
28.   procedure Group1_1;
29.   begin
30.     Write('Enum_All := TEnum_All(IntValue)');
31.     Enum_All := specialize safe_enum<TEnum_All>(IntValue);
32.   end;
33.
34.   procedure Group1_2;
35.   begin
36.     Write('Enum_1_3 := TEnum_All(IntValue)');
37.     Enum_1_3 := specialize safe_enum<TEnum_All>(IntValue);
38.   end;
39.
40.   procedure Group1_3;
41.   begin
42.     Write('Enum_0_4 := TEnum_All(IntValue)');
43.     Enum_0_4 := specialize safe_enum<TEnum_All>(IntValue);
44.   end;
45.
46.   procedure Group2_1;
47.   begin
48.     Write('Enum_All := TEnum_1_3(IntValue)');
49.     Enum_All := specialize safe_enum<TEnum_1_3>(IntValue);
50.   end;
51.
52.   procedure Group2_2;
53.   begin
54.     Write('Enum_1_3 := TEnum_1_3(IntValue)');
55.     Enum_1_3 := specialize safe_enum<TEnum_1_3>(IntValue);
56.   end;
57.
58.   procedure Group2_3;
59.   begin
60.     Write('Enum_0_4 := TEnum_1_3(IntValue)');
61.     Enum_0_4 := specialize safe_enum<TEnum_1_3>(IntValue);
62.   end;
63.
64.   procedure Group3_1;
65.   begin
66.     Write('Enum_All := TEnum_0_4(IntValue)');
67.     Enum_All := specialize safe_enum<TEnum_0_4>(IntValue);
68.   end;
69.
70.   procedure Group3_2;
71.   begin
72.     Write('Enum_1_3 := TEnum_0_4(IntValue)');
73.     Enum_1_3 := specialize safe_enum<TEnum_0_4>(IntValue);
74.   end;
75.
76.   procedure Group3_3;
77.   begin
78.     Write('Enum_0_4 := TEnum_0_4(IntValue)');
79.     Enum_0_4 := specialize safe_enum<TEnum_0_4>(IntValue);
80.   end;
81.
82.   procedure TestProc(ATestProc: TProcedure);
83.   begin
84.     try
85.       ATestProc;
86.       WriteLn(' Success');
87.     except
88.       on E: Exception do WriteLn(' Failed: ', E.Message);
89.     end;
90.   end;
91.
92. begin
93.   IntValue := 0; (* Integer IntValue is outside all enum ranges *)
94.
95.   Writeln('Group 1');
96.   TestProc(@Group1_1);
97.   TestProc(@Group1_2);
98.   TestProc(@Group1_3);
99.   WriteLn;
100.
101.   Writeln('Group 2');
102.   TestProc(@Group2_1);
103.   TestProc(@Group2_2);
104.   TestProc(@Group2_3);
105.   WriteLn;
106.
107.   Writeln('Group 3');
108.   TestProc(@Group3_1);
109.   TestProc(@Group3_2);
110.   TestProc(@Group3_3);
111.   WriteLn;
112.
114. end.

For a IntValue of 0 the output is:

Group 1
Enum_All := TEnum_All(IntValue) Success
Enum_1_3 := TEnum_All(IntValue) Failed: Range check error
Enum_0_4 := TEnum_All(IntValue) Success

Group 2
Enum_All := TEnum_1_3(IntValue) Failed: safe_enum range check error
Enum_1_3 := TEnum_1_3(IntValue) Failed: safe_enum range check error
Enum_0_4 := TEnum_1_3(IntValue) Failed: safe_enum range check error

Group 3
Enum_All := TEnum_0_4(IntValue) Success
Enum_1_3 := TEnum_0_4(IntValue) Failed: Range check error
Enum_0_4 := TEnum_0_4(IntValue) Success

IntValue = 1, you get:

Group 1
Enum_All := TEnum_All(IntValue) Success
Enum_1_3 := TEnum_All(IntValue) Success
Enum_0_4 := TEnum_All(IntValue) Success

Group 2
Enum_All := TEnum_1_3(IntValue) Success
Enum_1_3 := TEnum_1_3(IntValue) Success
Enum_0_4 := TEnum_1_3(IntValue) Success

Group 3
Enum_All := TEnum_0_4(IntValue) Success
Enum_1_3 := TEnum_0_4(IntValue) Success
Enum_0_4 := TEnum_0_4(IntValue) Success

And, an IntValue of 10 outputs:

Group 1
Enum_All := TEnum_All(IntValue) Failed: safe_enum range check error
Enum_1_3 := TEnum_All(IntValue) Failed: safe_enum range check error
Enum_0_4 := TEnum_All(IntValue) Failed: safe_enum range check error

Group 2
Enum_All := TEnum_1_3(IntValue) Failed: safe_enum range check error
Enum_1_3 := TEnum_1_3(IntValue) Failed: safe_enum range check error
Enum_0_4 := TEnum_1_3(IntValue) Failed: safe_enum range check error

Group 3
Enum_All := TEnum_0_4(IntValue) Failed: safe_enum range check error
Enum_1_3 := TEnum_0_4(IntValue) Failed: safe_enum range check error
Enum_0_4 := TEnum_0_4(IntValue) Failed: safe_enum range check error

Which is what I "expected" from my initial tests. ie: all of the castings should fail with IntValue 1234.

« Last Edit: July 30, 2022, 08:00:13 am by derek.john.evans »

derek.john.evans

• Jr. Member
• Posts: 55
Re: Range checking Enum types. Unreliable or confusing?
« Reply #3 on: July 30, 2022, 07:56:23 am »
So, my question is, does FreePascal or Delphi have a "safe_enum" style generic function hiding somewhere?

Code: Pascal  [Select][+][-]
1. generic function safe_enum<T>(AValue: integer): T;
2. begin
3.   if (AValue < Ord(Low(T))) or (AValue > Ord(High(T))) then begin
4.     raise ERangeError.Create('safe_enum range check error');
5.   end;
6.   Result := T(AValue);
7. end;

• Hero Member
• Posts: 11773
Re: Range checking Enum types. Unreliable or confusing?
« Reply #4 on: July 30, 2022, 10:19:02 am »
Enums ARE safe. Your code is simply wrong. And the compiler tells you very friendly....
Enums count from zero...
Ask again if you do not see the issue....
Black themes should be banned.

Jonas Maebe

• Hero Member
• Posts: 970
Re: Range checking Enum types. Unreliable or confusing?
« Reply #5 on: July 30, 2022, 10:32:03 am »
So, my question is, does FreePascal or Delphi have a "safe_enum" style generic function hiding somewhere?
Not as far as I'm aware. There is a plan to support casting integers to enums using the "as" operator in FPC though, which would be a safe typecast.

Regarding your original program, the reasons it doesn't work as you expect, are:
• An explicit typecast never performs any kind of range checking by design.
• When you read the value of a variable with type X, the compiler will always assume that this value is a valid type X value. That is what having a type system is all about. If it could not assume that, it would have to insert checks for every single assignment and use, even when assigning a variable to another variable of the same type.
This means that any kind of validity checking must be performed _before_ you explicitly cast a value to a particular type, because afterwards it is too late. Data may even get discarded by the explicit type cast, so the value seems be "valid" afterwards even if it was invalid originally. E.g.

Code: Pascal  [Select][+][-]
1. {\$packenum 1}
2. type
3.   tenum = (e1, e2, e3, e4);
4.   tpackedenumrec = bitpacked record
5.     a,b,c,d: tenum;
6.   end;
7.
8. var
9.   e: tenum;
10.   r: tpackedenumrec;
11. begin
12.   // e will contain 1 = e2 after this assignment, because the typecast discarded the upper byte. That is a valid tenum value.
13.   e:=tenum(\$101);
14.   // r.a will contain 2 = e3 after this assignment, because the bitpacking only allocates 2 bits per enum value in the record. That is a valid tenum value.
15.   r.a:=tenum(10);
16. end.
17.

As a result, forcing values outside the range of an enum into an enum variable means that any further use of that enum is considered to be undefined behaviour in FPC. Delphi behaves more like C/C++: there, storing values that do not fall outside the range of the underlying type used to hold the enum ("byte" in this case), is not considered to lead to an undefined result. Although it will still fail to trigger the range check errors on assignment in your test program (or if you use such variables for indexing arrays), so it's not fully supported either in terms of range checking. See https://www.mail-archive.com/fpc-devel@lists.freepascal.org/msg38722.html for more information.

• Hero Member
• Posts: 11773
Re: Range checking Enum types. Unreliable or confusing?
« Reply #6 on: July 30, 2022, 10:51:22 am »
Yes, but Jonas, the whole idea about enums and declaring a range on it is a predictable range and that simply work, with the provisions:
1. You do not hardcast, as you explained
2. You not use value members, like xxx=100
3. Enums count from zero, except see 2.
4. You do not use untyped pointers

In the latter case the range is expanded and the range can contain unnamed values.
We discussed that, I believe, because I attended that to you. (And a won't fix)
Option 3 is not an issue, the cast is.

And the unnamed members can be seen through RTTI if really needed in the case of 2.

Anyway, FPC handles it correctly.
« Last Edit: July 30, 2022, 10:55:47 am by Thaddy »
Black themes should be banned.

derek.john.evans

• Jr. Member
• Posts: 55
Re: Range checking Enum types. Unreliable or confusing?
« Reply #7 on: July 30, 2022, 12:17:43 pm »
Enums ARE safe. Your code is simply wrong. And the compiler tells you very friendly....
Enums count from zero...
Ask again if you do not see the issue....

"Your code is simply wrong"? Um, how so? I was expecting the compiler to raise a range error, and it didn't. The point of the exercise was to confirm which bad cast's would be detected by the compiler.

Since the compiler doesn't raise an exception when needed, that leaves me a number of choices:

1) Ignore the issue and hope the database integer fields always contain valid integers.
2) Write adhoc inline code everywhere there is a integer to enum casting. Which makes for a messy code base. EG:
Code: Pascal  [Select][+][-]
1. var
2.   I: integer;
3.   A: TMyEnum;
4.
5.   I := FIniFile.ReadInteger('settings', 'enum', 0);
6.   if (I >= ord(low(TMyEnum))) and (I <= ord(high(TMyEnum))) then begin
7.     A := TMyEnum(I);
8.   end;
3) Write integer to enum functions for each enumerated type I store in various tables: eg:
Code: Pascal  [Select][+][-]
1. function TMyEnumCast(A: integer): TMyEnum;
2. begin
3.   if (A >= ord(low(TMyEnum))) and (A <= ord(high(TMyEnum))) then begin
4.     Result := TMyEnum(A);
5.   end else begin
7.   end;
8. end;
4) or, use a generic function which mimics the feature C++ coders have had for some time. eg: static_cast
https://en.m.wikipedia.org/wiki/Static_cast

Code: Pascal  [Select][+][-]
1. generic function enum_cast<T>(AValue: integer): T;
2. begin
3.   if (AValue < Ord(Low(T))) or (AValue > Ord(High(T))) then begin
4.     raise ERangeError.Create('enum_cast range check error');
5.   end;
6.   Result := T(AValue);
7. end;
8.
9. generic function enum_cast<T>(AValue: integer; ADefault: T): T;
10. begin
11.   if (AValue < Ord(Low(T))) or (AValue > Ord(High(T))) then begin
13.   end else begin
14.     Result := T(AValue);
15.   end;
16. end;

I choose the latter. The aim of this topic was to make sure I haven't missed a new language feature before committing to a solution.
« Last Edit: July 30, 2022, 12:37:17 pm by derek.john.evans »

• Hero Member
• Posts: 11773
Re: Range checking Enum types. Unreliable or confusing?
« Reply #8 on: July 30, 2022, 12:48:29 pm »
Ranges are not a new language feature. It is well over 25 years old. I consider it basic (pun) Pascal.
Black themes should be banned.

nanobit

• Full Member
• Posts: 114
Re: Range checking Enum types. Unreliable or confusing?
« Reply #9 on: July 30, 2022, 01:02:06 pm »
For those interested in more background about FPC enums, my notes contain a summary.

derek.john.evans

• Jr. Member
• Posts: 55
Re: Range checking Enum types. Unreliable or confusing?
« Reply #10 on: July 30, 2022, 01:04:56 pm »
Ranges are not a new language feature. It is well over 25 years old. I consider it basic (pun) Pascal.
O, my god. No offense, but why would you say that? Very odd.

Lets cut to the chase. HiSpeed pascal from 1990 throws a range error with the following code. ....FreePascal does not.
Code: Pascal  [Select][+][-]
1. type TEnum = (a0, a1, a2, a3);
2. var i: integer; a: TEnum;
3. begin
4.   i := 100;
5.   a := TEnum(i);
6. end.

But!!! if I run this, I get a range error:
Code: Pascal  [Select][+][-]
1. type
2.   TEnum = (a0, a1, a2, a3);
3.   TEnumSafe = a0..a3;
4.
5. var i: integer; a: TEnum;
6. begin
7.   i := 100;
8.   a := TEnumSafe(i);
10. end.

I'd say that is dam interesting behavior. Not what I expected.
Hence this topic.

Martin_fr

• Hero Member
• Posts: 8099
• Debugger - SynEdit - and more
Re: Range checking Enum types. Unreliable or confusing?
« Reply #11 on: July 30, 2022, 01:13:52 pm »
Well you can cast to a compatible type

Code: Pascal  [Select][+][-]
1. type
2.    TEnum_All = (opt0, opt1, opt2, opt3, opt4);
3.    TEnum_All_CAST = type TEnum_All;
4. var
5.   a: TEnum_All;
6. begin
7.   a := TEnum_All_CAST(some_int_val);

But, that has a serious down side.

Consider "range check" as a means to find bugs in your app, i.e. errors that you neither know, nor expect.
An out of range value in the above cast, however is something that you do expect. (apparently, since you want error checking).

Now you wrap that code into a "try expect" block.
And if the "try expect" catches any range check, your code handles it, as some_int_val must have been wrong.
That works, if your "try Except" is really just covering that one single statement. (But then an "if .... then" is same as good).

The "try except" can handle the error further out in your code. But then it covers more code, and that extra code  may have had a genuine range check error somewhere else. And an unexpected range check some where else should not be treated equal to the expected error, because now some other variable has a "bad" value, and your entire app may be unstable.

If you keep range check for real bugs, then debugging will be a lot easier.
And in releases, well it should tell the user "save your work", because a crash may be imminent.

So, you should definitely use an "if ... then raise TEnumOutOfRangeFromDB.create(fieldname, bad_int_value, errormessage, other_important_data);"
And then your except should check for that exact error only.

Or if indeed you "try except" encloses only a single line => consider "if then" without any exception....

440bx

• Hero Member
• Posts: 2933
Re: Range checking Enum types. Unreliable or confusing?
« Reply #12 on: July 30, 2022, 01:18:07 pm »
@derek,

your confusion is quite understandable since there are subtleties involved.

Using the examples you provided in your previous post, TEnum is _not_ a range definition.  An enumeration is just a collection of named constants that belong to a group of constants called "TEnum".  Think of "TEnum" as a scope, not a range (a bit like a field in a record, the field is a member of the scope identified by the record name.)

In the second case, TEnumSafe is a range definition therefore the compiler can check the range.

The above said, different Pascal compilers can look at those definitions in different ways but, this is the way FPC has chosen to "look" at it and, it is not incorrect.  It's just one way it can be viewed.

HTH.
FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

Jonas Maebe

• Hero Member
• Posts: 970
Re: Range checking Enum types. Unreliable or confusing?
« Reply #13 on: July 30, 2022, 01:24:44 pm »
Lets cut to the chase. HiSpeed pascal from 1990 throws a range error with the following code. ....FreePascal does not.
Code: Pascal  [Select][+][-]
1. type TEnum = (a0, a1, a2, a3);
2. var i: integer; a: TEnum;
3. begin
4.   i := 100;
5.   a := TEnum(i);
6. end.
That's interesting. FPC models most of its behaviour on Turbo Pascal and Delphi, which do not throw a range check error in this case (an explicit type cast there means "do what I say, ignore any type information you may think you have"). My explanation was based on the assumption that this was "the normal/expected behaviour", but I see now that is not necessarily the case.

I wonder what UCSD Pascal did, which is what Turbo Pascal was based on.

Kays

• Sr. Member
• Posts: 429
• Whasup!?
Re: Range checking Enum types. Unreliable or confusing?
« Reply #14 on: July 31, 2022, 01:48:38 pm »
[…] HiSpeed pascal from 1990 throws a range error with the following code. ....FreePascal does not.
Code: Pascal  [Select][+][-]
1. type TEnum = (a0, a1, a2, a3);
2. var i: integer; a: TEnum;
3. begin
4.   i := 100;
5.   a := TEnum(i);
6. end.
Typecasting disables range checks, hence 100 can be converted into TEnum even though there is no such named value.

But!!! if I run this, I get a range error:
Code: Pascal  [Select][+][-]
1. type
2.   TEnum = (a0, a1, a2, a3);
3.   TEnumSafe = a0..a3;
4.
5. var i: integer; a: TEnum;
6. begin
7.   i := 100;
8.   a := TEnumSafe(i);