Welcome to a hopefully short but thorough example of importing data into a Pick system.
The goal is to get /etc/passwd and /etc/group and their respective shadows into Pick so we can store copies of the Linux users inside Pick. This way I have an easy to use script to update new machines when we migrate. Currently I have a shell script that gets the users from one system and copies them to the new system. However I'm a big believer in moving as much to inside Pick as possible.
Pick in this case is a shorthand to say any Pick like system like UniVerse, UniData, D3, OpenQM or ScarletDME. I'm writing this example in ScarletDME while actually planning to use this tool in UniVerse.
There is a github repo with all the code.
Forgive the pun.
The first step is to read /etc/passwd and print it to the screen.
*
OPENPATH '/etc' TO ETC.FILE ELSE
PRINT 'Failed to open: /etc'
STOP
END
*
READ PASSWD FROM ETC.FILE,'passwd' ELSE
PRINT 'Failed to read: /etc/passwd'
STOP
END
*
PRINT PASSWD
*
STOP
*
* END OF PROGRAM
*
END
*
The first step is to open the directory. Once we have a handler to the directory, we can then use a regular READ statement to read in the passwd file.
Pick does have ways of opening and reading files directly. You could use OPENSEQ and READSEQ with /etc/passwd to read the file but I prefer using the regular READ statement especially as I already know the Linux files are all plaintext.
If the files weren't plaintext, it would likely be better to use the sequential statements.
Once we have read the passwd file, we can then print it directly to the screen.
Now that we have a very simple program working, it's time to eat out vegetables.
We need to create the FILE that's going to hold our user data and set up our dictionary.
We first need to look at the structure of /etc/passwd. This will dictate how we create the dictionary.
We also need to look at /etc/shadow as I do want to store the password hashes as well.
There is a good article about the structure at cyberciti. You can also get the structure on the man pages.
The core structure is:
username:password:user_id:group_id:GECOS:directory:shell
Most of the fields are self-explanatory except for the GECOS. The GECOS field is a comment field that you store almost any information. It is free-form but usually it will hold extra information in a comma delimited way.
The password field will contain an x for most users as the real passwords are stored in the shadow file. This implies that at one point passwords may have been stored as plaintext in the passwd file. I'll need to do research on that to see if that's true.
Now for shadow, the core structure of it is:
username:password:last_changed:min_age:max_age:warning_period:inactivity_period:expiry_date
These fields are a bit more nuanced as they control different aspects of the password.
These are the fields you would modify to control how long passwords last, when they fully expire, when warnings to be shown and etc.
You can read more in the man page.
I want to store all the fields in pick so the first thing is to convert the above names to the Pick style:
USERNAME
PASSWORD
USER.ID
GROUP.ID
GECOS
DIRECTORY
SHELL
LAST.CHANGED
MIN.AGE
MAX.AGE
WARNING.PERIOD
INACTIVITY.PERIOD
EXPIRY.DATE
The PASSWORD field will hold the hash from shadow. Otherwise all the other fields willbe saved.
Now that we know the structure of passwd and shadow and also defined the fields in the Pick style. Now we can create the file and set up the dictionaries.
I wrote a utility called ADD-DICT that lets you add dictionaries from TCl without editing the raw records.
I also usually use this utility inside a BASIC program so that setting up a file is a bit easier.
But first, let's create the file:
CREATE-FILE LINUX-USER-FILE
This will work in ScarletDME but for UniVerse you will need to supply some extra information:
CREATE-FILE LINUX-USER-FILE 1,11 11,1
I have NEW-FILE command that abstracts these extra parameters away and unifies creating files across the different systems.
NEW-FILE LINUX-USER-FILE
Now that we have the FILE created, we can now set up the dictionaries.
Create the following program called LINUX.USER.SCHEMA:
*
EXECUTE 'CREATE-FILE LINUX-USER-FILE'
*
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 1 -N PASSWORD'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 2 -J R -N USER.ID'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 3 -J R -N GROUP.ID'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 4 -N GECOS'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 5 -N DIRECTORY'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 6 -N SHELL'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 7 -N LAST.CHANGED'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 8 -N MIN.AGE'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 9 -N MAX.AGE'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 10 -N WARNING.PERIOD'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 11 -N INACTIVITY.PERIOD'
EXECUTE 'ADD-DICT -F LINUX-USER-FILE -A 12 -N EXPIRY.DATE'
*
STOP
*
* END OF PROGRAM.
*
END
*
This acts as a migration script to update the dictionaries. This script will set up all the dictionaries. These are all relatively simple entries as we dont have numbers or real dates. We will store everything as it was stored in the Linux files.
We don't have the username as a dictionary item because the username will be the key to the record.
We use the -J option to set the justification. The USER.ID and GROUP.ID are numeric and I want to be able to sort on them. This means that we need to make sure the justification is set. If we had dates or if the numbers needed some conversion, we could also specific the conversion by using -CV.
Now with the structure studied and the file created, we can import some records.
The import is going to be simple because the Linux files are plaintext and use : as delimiters, it's already very analogous to the Pick way of doing things. Even the GECOS field which uses commas as delimiters means that we can convert those to regular value marks.
*
EQU TRUE TO 1
EQU FALSE TO 0
*
* COMPILER DIRECTIVES
*
$DEFINE DATABASE.QM
$DEFINE PLATFORM.LINUX
*
$IFDEF DATABASE.QM
$CATALOGUE LOCAL
$ENDIF
*
* %INCLUDE LINUX-USER-FILE
*
OPEN '','LINUX-USER-FILE' TO LINUX.USER.FILE ELSE
PRINT 'Failed to open file: LINUX-USER-FILE'
STOP
END
*
DIM LINUX.USER.ITEM(12)
MAT LINUX.USER.ITEM = ''
*
EQU LINUX.USER.PASSWORD.ATTRIBUTE TO 1
EQU LINUX.USER.USER.ID.ATTRIBUTE TO 2
EQU LINUX.USER.GROUP.ID.ATTRIBUTE TO 3
EQU LINUX.USER.GECOS.ATTRIBUTE TO 4
EQU LINUX.USER.DIRECTORY.ATTRIBUTE TO 5
EQU LINUX.USER.SHELL.ATTRIBUTE TO 6
EQU LINUX.USER.LASt.CHANGED.ATTRIBUTE TO 7
EQU LINUX.USER.MIN.AGE.ATTRIBUTE TO 8
EQU LINUX.USER.MAX.AGE.ATTRIBUTE TO 9
EQU LINUX.USER.WARNING.PERIOD.ATTRIBUTE TO 10
EQU LINUX.USER.INACTIVITY.PERIOD.ATTRIBUTE TO 11
EQU LINUX.USER.EXPIRY.DATE.ATTRIBUTE TO 12
*
@USER1 = 'IMPORT.LINUX.USERS'
@USER2 = 'IMPORT.LINUX.USERS'
*
* %END
*
OPENPATH '/etc' TO ETC.FILE ELSE
PRINT 'Failed to open: /etc'
STOP
END
*
READ PASSWD FROM ETC.FILE,'passwd' ELSE
PRINT 'Failed to read: /etc/passwd'
STOP
END
*
The first half of the program should look familiar. It's the same OPEN and READ statements from before but now we have some constants also set up.
This will make the program much easier to read.
The big thing of note is that I use ITEM to mark dimensioned arrays and use the equates to access the fields within each item.
This makes things clear about what fields are being set and what each field actually means.
Now for the core logic:
NUMBER.OF.LINES = DCOUNT(PASSWD,@AM)
*
FOR I = 1 TO NUMBER.OF.LINES
LINE = PASSWD<I>
*
CONVERT ':' TO @AM IN LINE
CONVERT ',' TO @VM IN LINE
*
USERNAME = LINE<1>
*
MAT LINUX.USER.ITEM = ''
*
LINUX.USER.ITEM(LINUX.USER.USER.ID.ATTRIBUTE) = LINE<3>
LINUX.USER.ITEM(LINUX.USER.GROUP.ID.ATTRIBUTE) = LINE<4>
LINUX.USER.ITEM(LINUX.USER.GECOS.ATTRIBUTE) = LINE<5>
LINUX.USER.ITEM(LINUX.USER.DIRECTORY.ATTRIBUTE) = LINE<6>
LINUX.USER.ITEM(LINUX.USER.SHELL.ATTRIBUTE) = LINE<7>
*
PRINT 'Importing: ' : USERNAME
MATWRITE LINUX.USER.ITEM ON LINUX.USER.FILE, USERNAME
NEXT I
*
STOP
*
* END OF PROGRAM
*
END
*
The READ statement gave use the passwd file contents. The READ statement automatically converts newlines to attribute marks. This is why we can count the number lines by counting the attribute marks.
The next step is to loop through the lines. We convert the : and , to attribute marks and value marks. This way we can then reference parts of the line numerically.
We then set up the LINUX.USER.ITEM and populate it with the values.
If I had wanted to be more explicit, I would set up equates for the passwd structure as well.
Something like:
EQU PASSWD.USERNAME.ATTRIBUTE TO 1
so that I can do:
LINE<PASSWD.USERNAME.ATTRIBUTE>
For our purposes though, I think it's fine to just go numerically as there aren;t too many fields. If there were more or if the program was much longer, I'd set up the equates.
There is nothing worse than being 500 lines deep and not remembering what field 47 means and how it relates to field 92.
Now we can run the program and we should have imported all of the users from `/etc/passwd.
If we do a listing of the LINUX-USER-FILE, we should see the following:
> LIST LINUX-USER-FILE username
LINUX-USER-F : username
PASSWORD.... :
USER.ID..... : 1000
GROUP.ID.... : 1000
GECOS....... :
DIRECTORY... : /home/username
SHELL....... : /usr/bin/fish
LASt.CHANGED :
MIN.AGE..... :
MAX.AGE..... :
WARNING.PERI :
INACTIVITY.P :
EXPIRY.DATE. :
With that we have the first file being imported.
Now let's finish it off by importing in /etc/shadow. One thing to note is that only root can read shadow so you will need to run the BASIC program as root actually import shadow.
READ SHADOW FROM ETC.FILE,'shadow' ELSE
PRINT 'Failed to read: /etc/shadow'
STOP
END
*
NUMBER.OF.LINES = DCOUNT(SHADOW,@AM)
*
FOR I = 1 TO NUMBER.OF.LINES
LINE = SHADOW<I>
*
CONVERT ':' TO @AM IN LINE
CONVERT ',' TO @VM IN LINE
*
USERNAME = LINE<1>
*
MATREADU LINUX.USER.ITEM FROM LINUX.USER.FILE, USERNAME THEN
*
LINUX.USER.ITEM(LINUX.USER.PASSWORD.ATTRIBUTE) = LINE<2>
LINUX.USER.ITEM(LINUX.USER.LAST.CHANGED.ATTRIBUTE) = LINE<3>
LINUX.USER.ITEM(LINUX.USER.MIN.AGE.ATTRIBUTE) = LINE<4>
LINUX.USER.ITEM(LINUX.USER.MAX.AGE.ATTRIBUTE) = LINE<5>
LINUX.USER.ITEM(LINUX.USER.WARNING.PERIOD.ATTRIBUTE) = LINE<6>
LINUX.USER.ITEM(LINUX.USER.INACTIVITY.PERIOD.ATTRIBUTE) = LINE<7>
LINUX.USER.ITEM(LINUX.USER.EXPIRY.DATE.ATTRIBUTE) = LINE<8>
*
PRINT 'Importing shadow: ' : USERNAME
MATWRITE LINUX.USER.ITEM ON LINUX.USER.FILE, USERNAME
*
END ELSE
RELEASE LINUX.USER.FILE, USERNAME
PRINT 'Failed to read user: ' : USERNAME
STOP
END
NEXT I
*
We first read in the shadow file and count of the number of lines.
The difference here is that we attempt to read the user from the LINUX.USER.FILE as it should already be there due to us having already imported passwd.
We read in the record with MATREADU so that we hold the update lock, this way no one can write to the record while we are working on it.
We populate the remaining fields using the shadow entry and then we write it out.
If for some reason the record doesn't already exist, then we do a RELEASE statement to release the lock we grabbed and then we do a hard stop. This will ultimately be a deeper investigation as it means passwd and shadow aren't in sync.
After this we can do another listing:
LINUX-USER-F : username
PASSWORD.... : $y$j9T$O3u1/916rmY/S9YdmX
Sa1a$aKaTafTad25aooW3Qr1k
mahvAiKgYwr/QGVvb8LDjtA
USER.ID..... : 1000
GROUP.ID.... : 1000
GECOS....... :
DIRECTORY... : /home/username
SHELL....... : /usr/bin/fish
LAST.CHANGED : 20314
MIN.AGE..... : 0
MAX.AGE..... : 99999
WARNING.PERI : 7
INACTIVITY.P :
EXPIRY.DATE. :
With that we are done! We have now imported the passwd and shadow files into Pick.
We can add some smarts to clear the file and re-import things or we can leave it as it is and keep track of users that get removed from Linux.
We could run this routine everytime the Linux files change. However I'm going to keep it as a manual process.
> CLEAR-FILE DATA LINUX-USER-FILE
> RUN BP IMPORT.LINUX.USERS
With that done, the next post will cover exporting data from the LINUX-USER-FILE.