Solarian Programmer

My programming ramblings

C Programming - read a file line by line with fgets and getline, implement a portable getline version

Posted on April 3, 2019 by Paul

In this article, I will show you how to read a text file line by line in C using the standard C function fgets and the POSIX getline function. At the end of the article, I will write a portable implementation of the getline function that can be used with any standard C compiler.

Reading a file line by line is a trivial problem in many programming languages, but not in C. The standard way of reading a line of text in C is to use the fgets function, which is fine if you know in advance how long a line of text could be.

You can find all the code examples and the input file at the GitHub repo for this article.

Let’s start with a simple example of using fgets to read chunks from a text file. :

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 
 4 int main(void) {
 5     FILE *fp = fopen("lorem.txt", "r");
 6     if(fp == NULL) {
 7         perror("Unable to open file!");
 8         exit(1);
 9     }
10 
11     char chunk[128];
12 
13     while(fgets(chunk, sizeof(chunk), fp) != NULL) {
14         fputs(chunk, stdout);
15         fputs("|*\n", stdout);  // marker string used to show where the content of the chunk array has ended
16     }
17 
18     fclose(fp);
19 }

For testing the code I’ve used a simple dummy file, lorem.txt. This is a piece from the output of the above program on my machine:

 1 ~ $ clang -std=c17 -Wall -Wextra -pedantic t0.c -o t0
 2 ~ $ ./t0
 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 4 |*
 5 Fusce dignissim facilisis ligula consectetur hendrerit. Vestibulum porttitor aliquam luctus. Nam pharetra lorem vel ornare cond|*
 6 imentum.
 7 |*
 8 Praesent et nunc at libero vulputate convallis. Cras egestas nunc vitae eros vehicula hendrerit. Pellentesque in est et sapien |*
 9 dignissim molestie.
10 |*

The code prints the content of the chunk array, as filled after every call to fgets, and a marker string.

If you watch carefully, by scrolling the above text snippet to the right, you can see that the output was truncated to 127 characters per line of text. This was expected because our code can store an entire line from the original text file only if the line can fit inside our chunk array.

What if you need to have the entire line of text available for further processing and not a piece of line ? A possible solution is to copy or concatenate chunks of text in a separate line buffer until we find the end of line character.

Let’s start by creating a line buffer that will store the chunks of text, initially this will have the same length as the chunk array:

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 
 5 int main(void) {
 6     FILE *fp = fopen("lorem.txt", "r");
 7     // ...
 8 
 9     char chunk[128];
10 
11     // Store the chunks of text into a line buffer
12     size_t len = sizeof(chunk);
13     char *line = malloc(len);
14     if(line == NULL) {
15         perror("Unable to allocate memory for the line buffer.");
16         exit(1);
17     }
18 
19     // "Empty" the string
20     line[0] = '\0';
21 
22     // ...
23 
24 }

Next, we are going to append the content of the chunk array to the end of the line string, until we find the end of line character. If necessary, we’ll resize the line buffer:

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 
 5 int main(void) {
 6     // ...
 7 
 8     // "Empty" the string
 9     line[0] = '\0';
10 
11     while(fgets(chunk, sizeof(chunk), fp) != NULL) {
12         // Resize the line buffer if necessary
13         size_t len_used = strlen(line);
14         size_t chunk_used = strlen(chunk);
15 
16         if(len - len_used < chunk_used) {
17             len *= 2;
18             if((line = realloc(line, len)) == NULL) {
19                 perror("Unable to reallocate memory for the line buffer.");
20                 free(line);
21                 exit(1);
22             }
23         }
24 
25         // Copy the chunk to the end of the line buffer
26         strncpy(line + len_used, chunk, len - len_used);
27         len_used += chunk_used;
28 
29         // Check if line contains '\n', if yes process the line of text
30         if(line[len_used - 1] == '\n') {
31             fputs(line, stdout);
32             fputs("|*\n", stdout);
33             // "Empty" the line buffer
34             line[0] = '\0';
35         }
36     }
37 
38     fclose(fp);
39     free(line);
40 
41     printf("\n\nMax line size: %zd\n", len);
42 }

Please note, that in the above code, every time the line buffer needs to be resized its capacity is doubled.

This is the result of running the above code on my machine. For brevity, I kept only the first lines of output:

 1 ~ $ clang -std=c17 -Wall -Wextra -pedantic t1.c -o t1
 2 ~ $ ./t1
 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 4 |*
 5 Fusce dignissim facilisis ligula consectetur hendrerit. Vestibulum porttitor aliquam luctus. Nam pharetra lorem vel ornare condimentum.
 6 |*
 7 Praesent et nunc at libero vulputate convallis. Cras egestas nunc vitae eros vehicula hendrerit. Pellentesque in est et sapien dignissim molestie.
 8 |*
 9 Aliquam erat volutpat. Mauris dignissim augue ac purus placerat scelerisque. Donec eleifend ut nibh eu elementum.
10 |*

You can see that, this time, we can print full lines of text and not fixed length chunks like in the initial approach.

Let’s modify the above code in order to print the line length instead of the actual text:

 1 // ...
 2 
 3 int main(void) {
 4     // ...
 5 
 6     while(fgets(chunk, sizeof(chunk), fp) != NULL) {
 7 
 8     	// ...
 9 
10         // Check if line contains '\n', if yes process the line of text
11         if(line[len_used - 1] == '\n') {
12             printf("line length: %zd\n", len_used);
13             // "Empty" the line buffer
14             line[0] = '\0';
15         }
16     }
17 
18     fclose(fp);
19     free(line);
20 
21     printf("\n\nMax line size: %zd\n", len);
22 }

This is the result of running the modified code on my machine:

 1 ~ $ clang -std=c17 -Wall -Wextra -pedantic t1.c -o t1
 2 ~ $ ./t1
 3 line length: 57
 4 line length: 136
 5 line length: 147
 6 line length: 114
 7 line length: 112
 8 line length: 95
 9 line length: 62
10 line length: 1
11 line length: 428
12 line length: 1
13 line length: 460
14 line length: 1
15 line length: 834
16 line length: 1
17 line length: 821
18 
19 
20 Max line size: 1024

In the next example, I will show you how to use the getline function available on POSIX systems like Linux, Unix and macOS. Microsoft Visual Studio doesn’t have an equivalent function, so you won’t be able to easily test this example on a Windows system. However, you should be able to test it if you are using Cygwin or Windows Subsystem for Linux.

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 4 
 5 int main(void) {
 6     FILE *fp = fopen("lorem.txt", "r");
 7     if(fp == NULL) {
 8         perror("Unable to open file!");
 9         exit(1);
10     }
11 
12     // Read lines using POSIX function getline
13     // This code won't work on Windows
14     char *line = NULL;
15     size_t len = 0;
16 
17     while(getline(&line, &len, fp) != -1) {
18         printf("line length: %zd\n", strlen(line));
19     }
20 
21     printf("\n\nMax line size: %zd\n", len);
22 
23     fclose(fp);
24     free(line);     // getline will resize the input buffer as necessary
25                     // the user needs to free the memory when not needed!
26 }

Please note, how simple is to use POSIX’s getline versus manually buffering chunks of line like in my previous example. It is unfortunate that the standard C library doesn’t include an equivalent function.

When you use getline, don’t forget to free the line buffer when you don’t need it anymore. Also, calling getline more than once will overwrite the line buffer, make a copy of the line content if you need to keep it for further processing.

This is the result of running the above getline example on a Linux machine:

 1 ~ $ clang -std=gnu17 -Wall -Wextra -pedantic t2.c -o t2
 2 ~ $ ./t2
 3 line length: 57
 4 line length: 136
 5 line length: 147
 6 line length: 114
 7 line length: 112
 8 line length: 95
 9 line length: 62
10 line length: 1
11 line length: 428
12 line length: 1
13 line length: 460
14 line length: 1
15 line length: 834
16 line length: 1
17 line length: 821
18 
19 
20 Max line size: 960

It is interesting to note, that for this particular case the getline function on Linux resizes the line buffer to a max of 960 bytes. If you run the same code on macOS the line buffer is resized to 1024 bytes. This is due to the different ways in which getline is implemented on different Unix like systems.

As mentioned before, getline is not present in the C standard library. It could be an interesting exercise to implement a portable version of this function. The idea here is not to implement the most performant version of getline, but rather to implement a simple replacement for non POSIX systems.

We are going to take the above example and replace the POSIX’s getline version with our own implementation, say my_getline. Obviously, if you are on a POSIX system, you should use the version provided by the operating system, which was tested by countless users and tuned for optimal performance.

The POSIX getline function has this signature:

1 ssize_t getline(char **restrict lineptr, size_t *restrict n, FILE *restrict stream);

Since ssize_t is also a POSIX defined type, usually a 64 bits signed integer, this is how we are going to declare our version:

1 int64_t my_getline(char **restrict line, size_t *restrict len, FILE *restrict fp);

In principle we are going to implement the function using the same approach as in one of the above examples, where I’ve defined a line buffer and kept copying chunks of text in the buffer until we found the end of line character:

 1 // This will only have effect on Windows with MSVC
 2 #ifdef _MSC_VER
 3     #define _CRT_SECURE_NO_WARNINGS 1
 4     #define restrict __restrict
 5 #endif
 6 
 7 #include <stdio.h>
 8 #include <stdlib.h>
 9 #include <string.h>
10 #include <stdint.h>
11 #include <errno.h>
12 
13 /*
14 POSIX getline replacement for non-POSIX systems (like Windows)
15 Differences:
16     - the function returns int64_t instead of ssize_t
17     - does not accept NUL characters in the input file
18 Warnings:
19     - the function sets EINVAL, ENOMEM, EOVERFLOW in case of errors. The above are not defined by ISO C17,
20     but are supported by other C compilers like MSVC
21 */
22 int64_t my_getline(char **restrict line, size_t *restrict len, FILE *restrict fp) {
23     // Check if either line, len or fp are NULL pointers
24     if(line == NULL || len == NULL || fp == NULL) {
25         errno = EINVAL;
26         return -1;
27     }
28     
29     // Use a chunk array of 128 bytes as parameter for fgets
30     char chunk[128];
31 
32     // Allocate a block of memory for *line if it is NULL or smaller than the chunk array
33     if(*line == NULL || *len < sizeof(chunk)) {
34         *len = sizeof(chunk);
35         if((*line = malloc(*len)) == NULL) {
36             errno = ENOMEM;
37             return -1;
38         }
39     }
40 
41     // "Empty" the string
42     (*line)[0] = '\0';
43 
44     while(fgets(chunk, sizeof(chunk), fp) != NULL) {
45         // Resize the line buffer if necessary
46         size_t len_used = strlen(*line);
47         size_t chunk_used = strlen(chunk);
48 
49         if(*len - len_used < chunk_used) {
50             // Check for overflow
51             if(*len > SIZE_MAX / 2) {
52                 errno = EOVERFLOW;
53                 return -1;
54             } else {
55                 *len *= 2;
56             }
57             
58             if((*line = realloc(*line, *len)) == NULL) {
59                 errno = ENOMEM;
60                 return -1;
61             }
62         }
63 
64         // Copy the chunk to the end of the line buffer
65         memcpy(*line + len_used, chunk, chunk_used);
66         len_used += chunk_used;
67         (*line)[len_used] = '\0';
68 
69         // Check if *line contains '\n', if yes, return the *line length
70         if((*line)[len_used - 1] == '\n') {
71             return len_used;
72         }
73     }
74 
75     return -1;
76 }

This is how we can use the above function, for simplicity I kept the code and the function definition in the same file:

 1 // ...
 2 
 3 int64_t my_getline(char **restrict line, size_t *restrict len, FILE *restrict fp) {
 4 	// ...
 5 }
 6 
 7 int main(void) {
 8     FILE *fp = fopen("lorem.txt", "r");
 9     if(fp == NULL) {
10         perror("Unable to open file!");
11         exit(1);
12     }
13 
14     // Read lines from a text file using our own a portable getline implementation
15     char *line = NULL;
16     size_t len = 0;
17 
18     while(my_getline(&line, &len, fp) != -1) {
19         // fputs(line, stdout);
20         // fputs("|*\n", stdout);
21         printf("line length: %zd\n", strlen(line));
22     }
23 
24     printf("\n\nMax line size: %zd\n", len);
25 
26     fclose(fp);
27     free(line);
28 }

The above code gives the same results as the code that uses the POSIX’s getline function:

 1 ~ $ clang -std=c17 -Wall -Wextra -pedantic t3.c -o t3
 2 ~ $ ./t3
 3 line length: 57
 4 line length: 136
 5 line length: 147
 6 line length: 114
 7 line length: 112
 8 line length: 95
 9 line length: 62
10 line length: 1
11 line length: 428
12 line length: 1
13 line length: 460
14 line length: 1
15 line length: 834
16 line length: 1
17 line length: 821
18 
19 
20 Max line size: 1024

I’ve also tested the code with the Microsoft C compiler and gives identical results for the same input file.

If you want to learn more about C99/C11 I would recommend reading 21st Century C: C Tips from the New School by Ben Klemens:

or the classic C Bible, The C Programming Language by B.W. Kernighan, D.M. Ritchie:


Show Comments