Seems to me there is a bug in Sybase IQ 16.0. If the IQ UNIQUE value in an DDL statement is too big, you'll receive this error: All IQ large memory has been used, allocation canceled. Based on my experiments the maximal allowed value is 33554432 (decimal), which is 0x2000000 (hexadecimal). If you really need to specify IQ UNIQUE for a column with a huge cardinality exceeding this value (which I doubt), you can use just this one.

Additionally - looks to me that increasing of a new command-line parammeter -iqlm should cure problems with this error message. According to some Sybase document the memory parammeters -iqmc, -iqtc and -iqlm (main store cache, temp store cache and large memory) should be initially set 1:1:1.