内容目录
Data Mask
Mask Functions
| Function Name | Description | Arguments | Config Example | Old Data | Masked Data |
|---|---|---|---|---|---|
| LaplaceDPFloat64 | differential privacy masking based on laplace | 1: l0sensitivity, 2: lInfSensitivity, 3: epsilon, 4: delta | LaplaceDPFloat64,100,1,1,0 | 3200.53 | 3074.3990583359264 |
| LaplaceDPInt64 | differential privacy masking based on laplace | 1: l0sensitivity, 2: lInfSensitivity, 3: epsilon, 4: delta | LaplaceDPInt64,100,1,1,0.5 | 3200 | 3198 |
| FF1 | format-preserving encryption, ff1 algorithm | 1: radix(default 10, max size 62, min size 2, [0-9a-zA-Z]), 2: key, 3: tweak | FF1,10 | 123 | 009 |
| FF3 | format-preserving encryption, ff3 algorithm | 1: radix(default 10, max size 62, min size 2, [0-9a-zA-Z]), 2: key, 3: tweak | FF3,10 | 123 | 080 |
| Base64 | base64 encryption | no argument | Base64 | abc | YWJj |
| DES | des encryption | 1: key, 2: iv | DES | hello world | jZLr1ir1An0IQc30XbLL3A== |
| AES | aes encryption | 1: key, 2: iv | AES | hello world | F5QMumnZOlCchKi2nu99rA== |
| TDEA | tdea encryption | 1: key, 2: iv | TDEA | hello world | F0HMxhk+uKKBSlR1IAyt+Q== |
| AESCTR | aesctr encryption | 1: key, 2: iv | AESCTR | hello world | 0URdmXIvWjcZS3U= |
| RSA | rsa encryption | RSA | hello world | ... ... | |
| ECC | ecc encryption | ECC | hello world | ... ... | |
| Fake | generate different type of fake data | 1: type(name, address, license-plate, email, ssn, birthday, creditcard, url, number, uuid, ip, ipv6) | Fake,Name | Any string | Auto generate fake string |
| CRC32 | crc32 hash | no argument | CRC32 | abc | 352441c2 |
| MD5 | md5 hash | no argument | MD5 | abc | 900150983cd24fb0d6963f7d28e17f72 |
| SHA1 | sha1 hash | no argument | SHA1 | abc | a9993e364706816aba3e25717850c26c9cd0d89d |
| SHA2 | sha2 hash | no argument | SHA2 | abc | ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad |
| HMAC | hmac hash | 1: hash function. e.g., md5, sha1, sha2 | HMAC,md5,"pass" | hello world | 37c4d226765f06daa3ad91a6c33a5d3e |
| Mangle | desensitization with corpus | 1: corpus name, 2. secret | Mangle,"en_US","secret password" | hello world! | spars dream! |
| Phone | phone default mask function | Phone | 13000000123 | 130*****123 | |
| mail default mask function | zhangsan001@d18n.com | z**@***m | |||
| Username | username default mask function | Username | Dave Li | D** | |
| Domain | domain default mask function | Domain | example.com | e*****m | |
| CreditCard | credit card default mask function | CreditCard | 6227612145830440 | 62276121****0440 | |
| PersonalID | personal id default mask function | PersonalID | 110223700003697 | 110223*****3697 | |
| USCC | china unified social credit code default desensitize method | USCC | 71797173LM37QP0D4H | 717971****0D4H | |
| Age | age round | Age | 46 | 40 | |
| Birthday | birthday default desensitize method | Birthday | 2020-05-34 | NNNN-NN-NN | |
| IP | IP address desensitize method | IP | 192.168.0.1 | 127.0.0.1 | |
| Password | password default desensitize method | Password | asfa@12323ssda | ***** | |
| Salary | salary default desensitize method | Salary | 1350 | 1000 | |
| LicensePlate | licensePlate default desensitize method | LicensePlate | 鄂D71D44 | 鄂****4 | |
| OrganizationCode | organizationCode default desensitize method | OrganizationCode | 100000439 | 100****439 | |
| Shuffle | shuffle and keep data type | 1. corpus name. e.g., 0-9, Lower, Upper, Chinese | Shuffle | 1234567890 | 4802731596 |
| ShuffleRight | shuffle right and keep data type | 1. index | ShuffleRight,4 | 1234567890 | 1234731596 |
| ShuffleLeft | shuffle left and keep data type | 1. index | ShuffleLeft,4 | 1234567890 | 4802737890 |
| Rot | letter substitution with the Nth letter after it in the alphabet | 1. radix. e.g., 47, 18, 13, 5, 32768 | Rot,5 | 1234567890 | 6789012345 |
| Morse | morse code | no argument | Morse | def | .- -... -.-. |
| Caesar | caesar code | 1. index | Caesar,3 | abc | def |
| Smoke | replace every characters with mask | 1. replacement | Smoke,"*" | 123 | *** |
| SmokeLeft | replace left n characters with mask | 1. index, 2. replacement | SmokeLeft,2,"*" | 123 | **3 |
| ReserveLeft | reserve left n characters, smoke right | 1. index, 2. replacement | ReserveLeft,2,"*" | 123 | 12* |
| SmokeRight | replace right n characters with mask | 1. index, 2. replacement | SmokeRight,2,"*" | 123 | 1** |
| ReserveRight | reserve right n characters, smoke left | 1. index, 2. replacement | ReserveRight,2,"*" | 123 | *23 |
| SmokeMargin | replace margin n characters with mask | 1. index, 2. replacement | SmokeMargin,2,"*" | 123456 | 34 |
| ReserveMargin | reserve margin n characters, smoke inner | 1. index, 2. replacement | ReserveMargin,2,"*" | 123456 | 12**56 |
| SmokeOuter | replace left, right outer n characters with mask | 1. left index, 2. right index, 3. replacement | SmokeOuter,2,1,"*" | 123456 | *345 |
| ReserveOuter | reserve left, right outer n characters, smoke inner | 1. left index, 2. right index, 3. replacement | ReserveOuter,2,1,"*" | 123456 | 12***6 |
| SmokeInner | replace left, right inner n characters with mask | 1. left index, 2. right index, 3. replacement | SmokeInner,2,1,"*" | 123456 | 12***6 |
| ReserveInner | reserve left, right inner n characters, smoke outer | 1. left index, 2. right index, 3. replacement | ReserveInner,2,1,"*" | 123456 | *345 |
| SmokeCharLeft | mask left before specify char | 1. index, 2. replacement | SmokeCharLeft,"@","*" | zhangsan123@example.com | ***@example.com |
| SmokeCharRight | mask right after specify char | 1. index, 2. replacement | SmokeCharRight,"@","*" | zhangsan123@example.com | zhangsan123@*** |
| Replace | string replace | 1. old string, 2. replacement, 3. n(-1 for all) | Replace,2,"",-1,"" | 123 | 1*3 |
| RegexpReplace | string regexp replace | 1. regexp, 2. replacement | RegexpReplace,"[bc]","*" | abcdef | a**def |
| RegexpRandomReplace | regexp random data replace | 1. regexp, 2. min, 3. max | RegexpRandomReplace,"^1[3-9][d]{9}$",5,10 | 13782430405 | 13614299600 |
| Reverse | reserve string | no argument | Reverse | abc | cba |
| ToUpper | string to upper | no argument | ToUpper | abc | ABC |
| ToLower | string to lower | no argument | ToLower | ABC | abc |
| Const | replace string with const | 1. replacement | Const,"MASKED" | abc | MASKED |
| Number2Const | replace all number to 9 | 1. replacement, default 9 | Number2Const | (+086)130-1234-123 | (+999)999-9999-999 |
| Char2Const | replace [a-zA-Z] to N | 1. replacement, default N | Char2Const | abc-def | NNN-NNN |
| NumberFloor | number round, replace lower n num with 0 | 1. index | NumberFloor,2 | 123.23 | 100 |
| DateRound | date round | 1. data format, 2. accuracy: second, minute, hour(default), day, month, year | DateRound,hour | 2021-07-23 17:26:45 | 2021-07-23 17:00:00 |
| Dateformat | convert date format | 1.old date format 2.new date format | dateformat,"YYYY-MM-DD HH:mm:ss", "YYYY/MM/DD HH-mm-ss" | 2021-07-23 17:26:45 | 2021/07/23 17-26-45 |
| LoopMoveLeft | string loop move to left | 1. index | LoopMoveLeft,3 | abcdefg | defgabc |
| LoopMoveRight | string loop move to right | 1. index | LoopMoveRight,3 | abcdefg | efgabcd |
| TruncateLeft | truncate left n characters | 1. index | TruncateLeft,2 | abcdef | ef |
| TruncateRight | truncate right n characters | 1. index | TruncateRight,2 | abcdef | ab |
| Abbreviate | english words abbreviate | Abbreviate | strategy-limited | stg-ltd | |
| Initialism | english words initialism | Initialism | hello world | hw | |
| Numeronym | a number-based word | Numeronym | internationalization | i18n | |
| SM2 | SM2 algorithm | SM2 | hello world | ... ... | |
| SM3 | SM3 algorithm | SM3 | hello world | ... ... | |
| SM4 | SM4 algorithm | SM4 | hello world | b9b1742de155fe5720c0b8b1b95e3134 |
Rand Seed
--rand-seed flag is used for generating random keys or values. Its default value is the current Unix nano timestamp.
Corpus
Generate a list of words using your system dictionary.
aspell -d en dump master | aspell -l en expand > mask/corpus/mangle.en_USGenerate maskFuncs
~ $ make mask-typoMask JSON data
If sensitive data contains in TEXT or JSON data type, use json mask function mark the column first, and config json keys mask rules in mask config file as columns mask rules.
$ cat mask.csv
j,json
password,smoke
firstname,ReserveLeft,1
lastname,ReserveLeft,1
$ cat query.sql
select '{"Userinfo": {"Firstname": "Bruce", "Lastname": "John", "Password": "123456"} }' as j from dual
$ d18n --defaults-extra-file test/my.cnf --mask mask.csv --query query.sql
+--------------------------------------------------------------------------+
| J |
+--------------------------------------------------------------------------+
| {"Userinfo":{"Firstname":"B****","Lastname":"J***","Password":"******"}} |
+--------------------------------------------------------------------------+
