************************************* * * * DB/C Newsletter * * February 2000 * * * ************************************* News and Comments One more time: ALERT! There is a nasty bug in DB/C 9.0 and DB/C 9.1 with a datestamp of May 1998 or earlier. The problem is essentially that PAUSE and *W don't terminate on February 29, 2000. You can find out more about this in the FAQs at http://www.dbcsoftware.com/dbcfaq.html#feb29. The solution is to obtain the final release of DB/C 9.1. Over the years, we have worked hard to make all versions of DB/C operate in exactly the same manner. Trying to make file locking and exclusive opens work the same on Netware, NT and the various UNIXes has been hard. This month's article presents a technically detailed peek into the internals of how all of this works. don.wills@dbcsoftware.com ***************************************************************************** More Than You Ever Wanted To Know About File Locks And Exclusive Open This article provides a detailed description of the internals of DB/C file processing done by DB/C DX, DB/C JX, and DB/C FS. Most of what is discussed in this article also applies to earlier releases of DB/C. However, certain details (e.g. large file support) apply only to the current release of DB/C DX (release 10.0.4) and to the next release of DB/C FS (release 2.2). DB/C provides programmers with three mechanisms for synchronization in a multi-user environment. These mechanisms are generally known as FILEPI, record locking and open mode control. FILEPI is the oldest and most-used synchronization control. It is a short-duration lock for all of the records in a file. When one program has an active FILEPI, all other programs will wait in the FILEPI statement until each is granted exclusive access to that file. Record locks are the newest and least-used form of synchronization. They grant exclusive control to one program for a single record in the file. Record locks are generally of longer duration than FILEPI. Multiple record locks may be obtained at different times. Thus record locks are susceptible to deadlock. Programmers should use record locking very carefully. Open mode control is a form of synchronization that allows a program to control access to a file by all other programs for the entire time a file is open. This usually means for the duration of a program. Exclusive opens are used frequently. Read-only opens are used very little. Internally, the OPEN statement is implemented with a call to the open function that is provided by the operating system. For Windows (Win32), the name of that C function is 'CreateFile'. For UNIX its name is 'open'. The Win32 'CreateFile' function provides an option for access control that works exactly as is needed by OPEN with the EXCLUSIVE or READ operand. Thus, the correct open mode functionality was easy to implement in the Windows version of each of the DB/C products. Specifically, in Win32, a shared mode OPEN uses the 'deny none' open option, an exclusive mode OPEN uses the 'deny read and write' option, and a read-only mode OPEN uses the 'deny write' option. A sharing mode option is not available in the UNIX open function. Thus, all UNIX opens are effectively shared mode opens. DB/C must emulate the exclusive and read-only OPEN modes in UNIX. It does this using UNIX file byte range locks as described in the following paragraphs. In addition to the three synchronization mechanisms available to DB/C programmers, there is one more file lock that is used internally by each DB/C product. There are several uses of this internal lock. Two examples are to protect end-of-file (-3) reads and writes, and to protect the logical integrity of index files (both ISAM and AIM) during read, write and delete manipulation of index information. All uses of an internal lock last only for the duration of a single DB/C statement. When a FILEPI is active for a file, the internal lock isn't necessary and thus it isn't done. FILEPI and the internal lock can therefore logically share the same locking method. Except for the Win32 open mode processing, all of these synchronization mechanisms are implemented using an operating system facility commonly called record locks or file byte range locks. We will use the term byte range lock to avoid confusion. Acquiring and freeing byte range locks is done by calling operating system functions with a file reference, the offset of the byte range, and the number of bytes in the range. In Win32, these functions are 'LockFile' and 'UnlockFile'. In UNIX, the 'fcntl' function provides both lock and unlock operations. A byte range lock may be acquired for any contiguous range of bytes in a file, including for a single byte. The start of that range is called its offset, which is the offset from the beginning of the file. The range of bytes isn't limited by the size of the file. That is, the range may be after the end-of-file. DB/C uses the principle of locking one byte at a high offset (beyond the end-of-file) to implement the synchronization mechanisms of DB/C. The reason for using a high offset is to prevent an error on those operating systems that fail during read and write when another user holds a byte range lock that includes the record being read and written. Netware is a prime example of an operating system that works this way. Here is a list of the offsets used for the three synchronization mechanisms (decimal values with equivalent hex values in parenthesis): Offsets of 1 byte range locks in Windows: FILEPI and internal file lock = 4294967280 (0xFFFFFFF0) record locks = 2147483648 (0x80000000) + offset of record to lock Offsets of 1 byte range locks in UNIXes with 32 bit file size support: FILEPI and internal file lock = 2147483632 (0x7FFFFFF0) record locks = 1073741824 (0x40000000) + offset of record to lock exclusive open = 2147483640 (0x7FFFFFF8) Offsets of 1 byte range locks in UNIXes with 64 bit file size support: FILEPI and internal file lock = 1099511627760 (0xFFFFFFFFF0) record locks = 549755813888 (0x8000000000) + offset of record to lock exclusive open = 1099511627768 (0xFFFFFFFFF8) With these values and the inherent file size limits, the following maximum file sizes are supported: Windows (Win32) file support allows for 2GB maximum file size when using record locks and 4GB maximum file size when not using record locks. UNIX with 32 bit file size support allows for 1GB maximum file size when using record locks and 2GB maximum file size when not using record locks. UNIX with 64 bit file size support allows for 500GB maximum file size when using record locks and 1TB maximum file size when not using record locks. These limits are actually based on the 40 bit limit for byte range offset that exists in HPUX. Other 64 bit UNIXes are capable of handling larger files. If for some reason, the default values for these 1 byte range locks need to be changed, DB/C DX 10.0.4 provides undocumented dbcdx.file properties to specify different values. Here are some examples of situations where the default values should be changed: 1. When you are not using record locks and you need to support larger files. For example, to support files that are greater than 1GB but smaller than 2GB in a 32 bit UNIX. 2. When trying to coordinate record locking with a non-DB/C product. 3. When mixing access from Windows and UNIX to a single server. For example, the Samba product allows Windows clients to access UNIX files as if they were on an NT server. Note that all DB/C clients that access a file need to use the same values for the byte range offsets. The dbcdx.file properties that change the byte range offsets need to be the same in all client environments. Contact us at support@dbcsoftware.com to get information about how to change these dbcdx.file properties. ***************************************************************************** Subscribing to the DB/C Newsletter If you don't already have the DB/C Newsletter delivered to your email address and would like to have it emailed to you when it is produced, just send an email message to 'request@dbcsoftware.com' and put the line 'subscribe dbcnews' in the body of the email message (omit the ' characters). The newsletter will be delivered to the email address from which the message was sent. To stop delivery, put the line 'unsubscribe dbcnews' in the body of the message.