We faced one challenge while creating Custom Links through Formula field using Hyperlink Function which should work inside Service Console (Behavior of Custom Links : Open in a new sub-tab window inside sales force application)
We got a solution after soo many work arounds :
HYPERLINK("javascript:if(typeof(srcUp)=='function') {srcUp('/URLDomainName?isdtp=vw'); } "+
"else {window.open='/URLDomainName?isdtp=vw'}","salesforceTab", "_parent")
you can use any url path/Apex Page inside srcUp function.
Above formula logic will open the Custom Links in a new sub-tab window inside the service Console and outside service console it will open in a new tab/window.
I have Posted this things in the below Community:
http://boards.developerforce.com/t5/Formulas-Validation-Rules/Hyperlink-in-Service-Cloud-Console-not-open-new-tab-or-subtab/td-p/309735
Additional Links which will help you in understanding :
http://success.salesforce.com/ideaView?id=08730000000YKxYAAW
Hope this post helps....:
